Point and click and describe – a lesson idea for engineering students

This lesson idea is based on what is called the Descriptive Camera, a camera which takes a picture and outputs a description of that picture.

Show students the following picture and say “Tell me something about this?”:

Follow up question – “What else can you say?”

Give them 3 minutes or so to respond. Write up on a board any engineering/interesting lexis.

Show them the next picture:

Ask them to label the above photo with the following:

  • Beaglebone(embedded Linux platform)
  • Thermal printer
  • Status LEDs
  • USB webcam

You could also elicit other electronic components seen in the photo e.g. power wire(red and black wires), signal wire (green, yellow, black wire), USB connector, power connector, Ethernet connector, breadboard.

Now divide students into two groups, A & B. Explain that each group will get a different text. Group A’s text will explain what the device is, why it was made and the results of the device. Group B’s text will describe how it works.

Group A text:

The Descriptive Camera works a lot like a regular camera—point it at subject and press the shutter button to capture the scene. However, instead of producing an image, this prototype outputs a text description of the scene. Modern digital cameras capture gobs of parsable metadata about photos such as the camera’s settings, the location of the photo, the date, and time, but they don’t output any information about the content of the photo. The Descriptive Camera only outputs the metadata about the content.

As we amass an incredible amount of photos, it becomes increasingly difficult to manage our collections. Imagine if descriptive metadata about each photo could be appended to the image on the fly—information about who is in each photo, what they’re doing, and their environment could become incredibly useful in being able to search, filter, and cross-reference our photo collections. Of course, we don’t yet have the technology that makes this a practical proposition, but the Descriptive Camera explores these possibilities.

After the shutter button is pressed, the photo is sent to Mechanical Turk for processing and the camera waits for the results. A yellow LED indicates that the results are still “developing” in a nod to film-based photo technology. With a HIT price of $1.25, results are returned typically within 6 minutes and sometimes as fast as 3 minutes. The thermal printer outputs the resulting text in the style of a polaroid print.

Matt Richardson, Descriptive Camera.

Group B text:

The technology at the core of the Descriptive Camera is Amazon’s Mechanical Turk API. It allows a developer to submit Human Intelligence Tasks (HITs) for workers on the internet to complete. The developer sets the guidelines for each task and designs the interface for the worker to submit their results. The developer also sets the price they’re willing to pay for the successful completion of each task. An approval and reputation system ensures that workers are incented to deliver acceptable results. For faster and cheaper results, the camera can also be put into “accomplice mode,” where it will send an instant message to any other person. That IM will contain a link to the picture and a form where they can input the description of the image.

The camera itself is powered by the BeagleBone, an embedded Linux platform from Texas Instruments. Attached to the BeagleBone is a USB webcam, a thermal printer from Adafruit, a trio of status LEDs and a shutter button. A series of Python scripts define the interface and bring together all the different parts from capture, processing, error handling, and the printed output. My mrBBIO module is used for GPIO control (the LEDs and the shutter button), and I used open-source command line utilities to communicate with Mechanical Turk. The device connects to the internet via ethernet and gets power from an external 5 volt source, but I would love to make a another version that’s battery operated and uses wireless data. Ideally, The Descriptive Camera would look and feel like a typical digital camera.

Matt Richardson, Descriptive Camera.

After each group has finished reading ask them to find someone from the other group to explain in their own words their text. Tell them that people from Group A should start the exchange. Also tell them that Group A will need to ask Group B to explain to them two things – the word Mechanical Turk and the abbreviation HIT.

Monitor and feedback as necessary.

Then get the groups to swap their text, each now reads the new text and writes 3 comprehension questions. The groups now find a +new+ person from the other group to ask the questions to.

Again monitor and feedback as necessary.

Various lexis could be followed up e.g. ask the students if they know what GPIO is and if they can point it out in the second photo above.

Additionally the following  video (up to the 3:44 mark) could be shown:

Example video comprehension questions: What additional reason did the inventor give for developing the prototype? What extra information did you hear from the video?

Various extensions could be done e.g. students can find out more about Amazon’s Mechanical Turk, the origins of the word.  Or a  discussion on whether students would buy such a device if commercialised. Or get students to describe the three photos shown at Descriptive Camera themselves.

Penny for your thoughts

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.