IntroductionEditAdvances in computing have brought about many innovative modalities when it comes to interacting with technology. From the lever since the 1900s to the touchscreen that is now ubiquitous amongst smart phones, humans have continually sought to reinvent the manner in which they communicate with technology. One of the most recent developments has been the emergence of a new type of interaction known as Organic User Interfaces (OUIs) .
OUIs are essentially the actual computer interface itself that can be touched or manipulated. It can be anything that is intuitive to humans, such as a mold that artists can interact with to form a 3D model on a computer. It is an exciting prospect that can transform the paradigm of human-computer interaction as we know it.
To that end, the Mixed Reality Lab team led by Jeffrey and Kasun has sought to explore the promise of an innovative OUI system based on ferroliquid . Ferroliquids are unique because they possess both the fluid properties of a liquid and the magnetic properties of a solid . Adding a twinge of artistry is the attribute of ferrofluids being able to form spikes along magnetic field lines when the magnetic surface force exceeds the combined effect of surface tension and fluid weight . In addition, ferrofluid changes its shape dynamically depending on strength and direction of the applied magnetic field.
These properties combined make ferrofluid an ideal medium for the interface system. It has been harnessed in many applications, including ferrofluid sculptures by Sachiko Kodama . Drawing inspiration from her, the team has embarked on a project called Liquid Interface. It entails the idea of using ferroliquid as a medium to achieve an input and output interface that responds in a manner that is interactive and fun for the user.
The setup so far comprises of a container of ferrofluid, an array of six electromagnets and Hall Effect sensors, alongside a wand with a neodymium magnet attached acting as the input instrument. The electromagnet creates a “buttons” that manifests as a series of spikes. It is analogous to half a porcupine submerged in fluid.
Each Hall Effect sensor is embedded beneath each “button” and above each electromagnet (insert photo). Sensing is achieved by using the Hall Effect to detect the presence of the wand when it is within close proximity to a “button”(insert another photo). More specifically, the Hall Effect sensor detects the magnetic density and invokes a response when a specific value of the magnetic density is reached. That value is set to be the point when the wand is directly above the “button” and Hall Effect sensor.
Three mechanisms work in conjunction to produce a visceral feedback to the end user.
- The first of which is the visual aspect realised by the electromagnet that morphs the ferrofluid “buttons” to a new shape when he/she positions the wand directly above any one of the “buttons”.
- The second mechanism is haptic feedback, and this is achieved by utilising the repulsion force between the like poles of the electromagnet and the neodymium magnet a force feedback vibration on the wand. Furthermore, by increasing the repulsion force during actuation, the system gains an additional layer of feedback.
- Rounding up the feedback effect is audio. In this current implementation of the setup, the “buttons” are programmed to play music notes. This is akin to playing a piano.
The overall effect is a liquid interface that is able to transform its shape dynamically based on user input. It is hoped that with this interface, we will be able to explore the use of liquids as an organic user interface. We dub this sensing modality, “Rare Earth Centric” sensing.
Aims and GoalsEdit
The next phase of the project entails conducting a user study. In this user study, we aim to compare two feedback modalities of Liquid Interface, rare-earth centric versus hall effect centric. The current setup achieves sensing via the use of a neodymium magnet that is attached to a wand and an array of six Hall Effect sensors overlayed on top of six electromagnets. A future implementation we dub “Hall Effect Centric” sensing will do away with the hall effect sensor array and the neodymium magnet wand. Instead, only one Hall Effect sensor will be used as the primary input device. It will be embedded in an attachment that can be worn easily on the hand, more likely in the form of a ring or a glove. Each of the six electromagnets will be set to a different frequency via a function generator. It is hoped that the system will be able to discern the different electromagnets through the use of the Hall Effect sensor, allowing us to achieve the same effect as the previous setup, sans the haptic feedback due to the lack of a repelling force between the electromagnet and the neodymium magnet on the wand in the first setup. The trade off is that we can achieve sensing that is not as messy as the first setup. This is because the ferrofluid have a tendency to be attracted the neodymium magnet on the wand, making the implementation a messy and uncomfortable experience for the user. A vibrating motor to be implemented in a wearable attachment, most likely a glove or a ring, will be used to simulate the haptic feedback instead. This implementation will offer us the flexibility to customise the haptic feedback to suit any application we come up with.
The user study will also allow us to explore several ideas the team have in mind and collect user feedback on the two systems such as the optimal setting for the feedback force the motor generates. In addition, the user study can help the team to discover errors and the areas of improvement that can be made. It is hoped we can implement the results from both systems in the next implementation of the Liquid Interface Project. In addition, we will be submitting our findings to three conferences, ISWC 2012, Pervasive 2012 and DIS 2012.
Summary of Progess, Work Done, Results Achieved and Future RoadmapEdit
The current aim is to test the hall effect sensor readings at different Power Wave Modulations (PWMs) of the electromagnet. The hall effect sensor must be able to discern the different electromagnets it is positioned above of. To that end, it is necessary to conduct experiments in order to accurately pinpoint the voltage values generated by the hall effect sensor at different PWMs. Once that is done, it will be possible to set the system to actuate the corresponding music note and electromagnet.
- Reviewed current progress of the system
- Read up on documentation pertaining to the project including the firmware and microcontroller datasheet as well as the papers that has been submitted so far
- Familiarised myself with the hardware
- Set up the hardware for the 2nd setup (Hall Effect Centric Sensing)
- Read through the C code that has been written so far for the ATMEGA 2560 microcontroller
- Programming the 2nd setup to realise rare earth centric sensing
- Came up with a few ways of interaction for the user study
- Conducted an experiment to determine how the magnetic field changes when all six electromagnets are turned on.
- Pick up Java Language and MAX MSP to be used to implement musical notes for the systems
I have replicated the setup my predecessor has completed and have begun to modify it to realise hall effect centric sensing. I have also conducted an experiment to determine the voltage outputs of the hall effect sensor when positioned directly above the electromagnets at different PWMs. On the programming side of things, I have written a program that is able to power up the electromagnets at different PWMs and takes in the analog values from the hall effect sensor. In addition, I have managed to program the system to discern between two electromagnets set to different PWMs, one at 200, the other at 150 with the hall effect sensor as the input device, as opposed to six hall effect sensor arrays in the previous setup.
The next step will be to program the system to discern between all six electromagnets with only one hall effect sensor. The system must not confuse the electromagnets from each other. This can prove to be difficult as the range of the PWMs is from 0 to 255. Setting too low of a PWM will not actuate the ferofluid enough to create a button and may result in uneven buttons to be formed instead. In addition, spreading the PWMs while ensuring that the difference in the physical actuation of the buttons on the ferofluid is not obvious may prove to be a challenging task. I will experiment with some ideas I have in mind such as reversing the polarity of alternate electromagnets so that I can achieve more distinct magnetic fields across all six magnets.
Once that is done, the next stage will be to incorporate the hall effect sensor into a wearable such as a glove or a ring. I will also need to figure out how to incorporate a motor into the wearable to simulate the haptic feedback in the 1st setup. This is because the loss of the neodymium magnet as the input mechanism also meant the loss of a repulsion force between like-poles between the neodymium and the electromagnets.
The third step would be to incorporate the results of the user study to come up with some form of interaction, most likely a game or a puzzle that is both intriguing and fun for the user.
To summarise, I will be
- Focusing on improving the method of detecting the different PWMs across all six electromagnets
- Implementing a wearable device for the input mechanism
- Figuring out the best way to incorporate haptic feedback using a motor
- Collecting user response from user study
- Designing some form of interaction such as games or puzzles for users
Video Demonstration of Sensing of Three Electromagnets
 Hiroshi Ishii, “The Tangible User Interface and its Evolution,”Communications of the ACM, vol. 51, June 2008.
 Jeffrey Tzu Kwan Valino Koh, “Liquid Interface: A Haptic, Ferrofluid-Based Organic User Interface”.
 Exploring the Nanoworld. (1993) Exploring the Nanoworld. [Online]. HYPERLINK “http://mrsec.wisc.edu/Edetc/background/ferrofluid/index.html” http://mrsec.wisc.edu/Edetc/background/ferrofluid/index.html
 R.F.Rosensweig M.D.Couley, “Fluid Mechanics,” pp. 671-688, 1967.
 Sachiko Kodama, “Dynamic ferrofluid sculpture,” Communications of the ACM, vol. 51, no. 6, June 2008. [Online]. HYPERLINK “http://www.organicui.org/?page_id=74” http://www.organicui.org/?page_id=74