Tuesday, August 21, 2007

DTN Additional Notes

An extra step required to make DTN configure after building is to add the following to the configure command in the berkeleydb.mk file in the buildroot packages directory:
--with-mutex=ARM/gcc-assembly \
After making this change, the db-* directory should be purged from the build_arm_nofpu directory to ensure proper rebuilding.

Without this change, DTN will will encounter the following error while initializing the db:
DB internal error: "unable to initialize mutex: Function not implemented"
Also when DTN is being built, it expects to find TCL header files in /usr/local/tcl-arm/include and TCL libraries in /usr/local/tcl-arm/lib. On the host system these should be linked to {buildroot_dir}/build_arm_nofpu/root/include and {buildroot_dir}/build_arm_nofpu/root/usr/lib respectviely.

After DTN is built, /lib/tcl8.4 should be linked to /usr/lib/tcl8.4, or during db initialization DTN will be very verbose about a misconfigured TCL installation.

Wednesday, August 15, 2007

DTN overview and instructions

DTN is a disaster tolerant network implementation. Some samples of how to use it and a short description may be found here. Over the past few weeks I've been trying to configure it to perform link state routing. I wanted to document all the things I had to set up to get discovery and routing to work, as well as any issues I've encountered with putting DTN on an arm platform.

To set it up, first add dtn2 package to buildroot. I took what Dave from prisms put together and adapted it to the new version of dtn. The package files can be grabbed here. Download the tarball, extract into ${gumstix-buildroot}/package and add dtn2 to the list of packages in Config.in. DTN depends on having a storage db and a tcl interpreter. Both need to be included in your buildroot make config before adding dtn to it. I found that building those first helps as well. The package files assume berkeleydb is the one installed. Also, the dtn package files assume that there is a local tcl build for arm in /usr/local/tcl-arm. You can change that by modifying the dtn2.mk file. Lastly, I forgot to amend the patch and it currently breaks. The fix is to add "-I/usr/local/tcl-arm/include" to LDFLAGS otherwise it won't be able to locate the tcl.h file. The file to modify for this is Rules.in in the ${dtn2}/oasys folder. After this dtn should build fine.

Once dtn is built, to configure it for dtlsr the following has to be done. For a simple scenario, you have to specify the interfaces and route type, as well as the node name. The interfaces are specified with "interface name type", e.g. interface udp0 udp or interface tcp0 tcp. This will create a listening socket on on the default port (4556). To specify the node name add 'route local_eid "name"' in the routing configuration. Another piece is to set the routing to use the dtlsr router. This can be done with 'route set type dtlsr'.

Lastly to use dtlsr the nodes need to send and listen for discoveries from other nodes. To set up the listening portion use the following command: 'discovery add discovery_name ip port=2000 local_addr='. The local_addr option is not required but useful if dtn should listen on only one interface. The port option is required and the ip type uses udp packets to broadcast. Now to add the announcements use the following: discovery announce name discovery_name tcp interval=5 cl_addr= This will send announcements using udp every 5 seconds on the interface with address If you have multiple interfaces, each of these has to be specified, i.e. you need an announcement for each with the interface ip address.

In order to change configuration of a running dtn instance, one can telnet to port 5050 on localhost and then execute all of the commands specified in the config file manually. One can also view the available routes, links, bundles, etc. To view the bundles use "bundle list". To view the routing table use "route dump".

This is sufficient to set up a simple dtn with link state routing. Often times you'd like to retrieve the data from this network. For that purpose one could define a static link to a receiving computer that has a static ip. To do that use this command:
link add NAME dtn.dtnrg.org:4556 ONDEMAND tcp. Also add a route for it with: route add dtn://eid/* linkNAME. For example, route add dtn://dtnrg.dtn/* NAME. This will setup a static link to dtn.dtnrg.org node on port 4556 and create a route for it, assuming the eid is dtn://dtnrg.dtn. The information sent can be viewed on the receiver with bundle list or use the dtnrecv command, which takes the destination dtn node eid and prints bundles addressed to it to screen.

Lastly, before running the dtn daemon for the first time you have to initialize the storage database. Do that with dtnd --init-db. Add "-c config_file" to use the "config_file". To set the storage, comment out
foreach dir {/var/dtn /var/tmp/dtn} {
if {[file isdirectory $dir]} {
set dbdir $dir
and specify the db with 'set dbdir "/home/dtn"' for example.

Monday, August 6, 2007

dtlsr and discovery

I got dtlsr to work properly, thanks to Mike Demmer for the pointers. In order to set it up one has to set route type to dtlsr and then it is important to also add discovery to the configuration. Here are the lines I used to set that up:
discovery add discovery_bonjour ip port=2000
discovery announce tcp0 discovery_bonjour tcp interval=5

They're pretty straightforward and there are more options for the commands I believe. In the end you should see node announcements broadcast every 5 seconds on port UDP 2000. You can verify if it's running by looking at a packet capture through wireshark. My next step is to test a simple three-four node network. The setup will involve a three wireless nodes: one with just one card in ad-hoc mode, acting as an end node; one with two cards on two different ad-hoc networks acting as a relay; lastly, the last gumstix node will be a gateway to a wired network where a destination node, my laptop is setup. I would like to test if dtn will handle traversing different subnets and also I'd like to see how long it takes to discover a path and how disconnected the network can be (essentially verify the paper on dtlsr: http://www.cs.berkeley.edu/~demmer/papers/dtlsr-nsdr.pdf)

Thursday, August 2, 2007

ad-hoc on prism2 with wlan-ng drivers

We were doing range tests with the antennae yesterday and I realized that I forgot how to put the prism2 cards with the wlan-ng driver in ad-hoc mode. I figured I'd put it here in case anyone else has that issue. Here is the command:
wlanctl-ng wlan0 dot11req_start ssid=gumstix bsstype=independent \
beaconperiod=100 dtimperiod=3 cfpollable=false cfpollreq=false cfpperiod=3 \
cfpmaxduration=100 probedelay=100 dschannel=11 basicrate1=2 \
basicrate2=4 operationalrate1=2 operationalrate2=4 operationalrate3=11
It should be run after wlanctl-ng wlan0 lnxreq_ifstate ifstate=enable. Sometimes the cards needs to be reset for this to work properly as well. Also, not all the options are required. For instance I believe the rates may be omitted.

I also have dtn running on the gumstix and I've been thinking of the best way to handle links, since we would want to do store and forward at every node, thus if packets are sent to the gateway they will not be stored on the hops in between. I learnt that there is a linkstate routing implementation for dtn. The router is named dtlsr. I will try to set it up and get working, since it seems exactly what we've been looking for.

Friday, July 27, 2007

RTC on the PXA270

In order to get DTN working properly in the future we need to have RTC setup and working on the gumstix. Jeremy and I at first thought that we could simply attach a backup battery to the V_BACKUP pin on the power control chip. We tried that and had 3V coming out on the V_RTC pin, but still the RTC would reset on every power down/restart. Then we thought it may be in software, so I looked through the driver code and commented out the line that resets the clock if the status register is misbehaving. The resets still persisted. Finally, I found out that on the PXA270 the RTC is on the chip and in order to keep the oscillator running we'd need to power the whole chip (at least keep it in deep sleep mode) and keep RAM alive. We decided to order an external RTC board, which is a DS1307 board which has I2C interface. Gumstix already has the patches for these boards and it should be pretty easy to get them setup once they come in and just set the PXA RTC to the external RTC.

Also, I've started to look at the CLA code for DTN2 and I'm still trying to figure out whether DTN sends out the bundles or if the CL has to do that. I should be able to understand that pretty soon. Ideally, I'll be able to simply notify DTN2 of link changes and have it handle message delivery. All the discovery can already be done by OLSR anyway.

Wednesday, July 25, 2007

more dtn on gumstix

I found that for some reason berkeleydb refuses to use pthreads installed on the gumstix. I've tried building the db with test and set by supplying "ARM/gcc-assembly". Also, I had to add -fPIC to CFLAGS to get it dtnd to load it on the verdex board. Lastly, it appears that the dtn build compiles in absolute paths, which has its issues since now it looks for the tcl in /home/timur/.... I ended up adding a symlink to the build. I will add more here once I make more progress.

Ok, Berkeley DB seems to be running with test-and-set, but now dtn crashes at the assertion: ASSERTION FAILED ((u_int)now.tv_sec >= TIMEVAL_CONVERSION) at bundling/BundleTimestamp.cc:39. I tried adding printf in there to see the value of now.tv_sec and I tried printing it to a file, but neither way produced any output (I did replace the dtnd file after recompiling). I'm not sure what the issue is and how to go about resolving it. I will look into this more tomorrow. I'm hoping Brian will know something about this, but not keeping my hopes up. If I can figure out why nothing is printed it'd be a big step in troubleshooting.

EDIT: Ok, the problem is not with the code or the assertion. DTN checks if the time is past midnight UTC December 31st, 1999 is (probably to make sure packet dates can be used correctly). The gumstix were set to midnight December 31st, 1969, thus the assertion was failing. I have DTN now running on the gumstix.

Monday, July 23, 2007

dtn and hostap

I built DTN for verdex with reasonable success and no big issues. For connex, using revision 1360 proved to be difficult so I tried a different one. Right now I'm trying to build revision 1340 with the packages required by dtn. One thing I have to fix is the path for berkeley db (which was also an issue with verdex). I think this will be done by tomorrow and then I can test it out.

On another note, with hostap it appears that ap_scan is very useful, as we can check ssids without invoking scan directly but instead examine the file in proc. This can be great in the future when we want passive scans to happen and simply check whether a new access point is available. I don't know if the linux-wlan driver supports this, but would be neat.

Justin has the communication setup and we looked into powering devices up/down. Looks like we can do it with wireless pretty easily by pccardctl and eject/insert commands. We're going to do more power measurements to verify that the card does indeed consume less power in that mode. I could not find a gpio pin to toggle to turn it off. Hopefully this will be sufficient.

Friday, July 20, 2007

building dtn

Dave's guide for DTN has been quite useful. A lot of the patches actually already appear in the later revision of the gumstix though. I had to add -ltcl8.4 to the LIBS in Rules.make in oasys to get liboasys to compile, thus getting dtn to compile. Everything else seems to compile fine.


Since AP and OLSR appear to be running as expected, I was looking into how to setup store and forward. It appears that the route to take is to try to setup DTN2. After reading RFC4838, I realized that DTN runs on top of the transport layer, which means that OLSR can still do the routing and a DTN daemon on top can handle broken/disconnected links and store the data if a link is not available. I found a few explanations as to how to set it up and I will try to have DTN setup today.

Wednesday, July 18, 2007

dhcpd and dyn_gw

I have setup DHCPd on the verdex board and it appears I can communicate with the board fine. I have one more test to see if I can setup a simple two hop network and check if I can see the dhcp clients on a different subnet, which are not running OLSR. Hopefully, it will work fine. For dynamic gateway operation I used the dyn_gw_plain plugin, which seems to work fine. We have to toggle the interface manually, i.e. bring it up and down, but it detects such changes. The script to scan for access points and verify connectivity seems to be working fine now as well.

Tuesday, July 17, 2007


Ok, Ming pointed out that running nodes in Ad-Hoc mode will make a lot more sense, since OLSR will take care of loops and can use signal strength to find the best route to a node. The script to scan for access points and associate with them was almost finished, but then I found linux-wlan-ng support wireless extensions if "options p80211 wlan_wext_write=1" is added to modprobe.conf. I will modify the script today and see if it works correctly. The next step is to get the dynamic gateway plugin to work. Last time I tried to use it, it segfaulted, but I have not looked into why too much. Worst case scenario I think I can put together a hack to do the same thing.

Friday, July 6, 2007

OLSR power consumption on gumstix

We set up a small circuit (thank you Matt) to measure the power consumption of the verdex board while running OLSR. We put a small, .5 Ohm resistor in series between the DC voltage generator and the board and measured the voltage drop. From my calculations the power consumed by the board at different stages is as follows:

Current Power
Compact flash in
0.04 0.08 0.42
Connected USB
0.05 0.09 0.46
Loaded the PCMCIA driver
0.04 0.08 0.39
Loaded the USB driver
0.04 0.08 0.39
Started AP
0.06 0.13 0.64
Associated with an AP. Started OLSRd
0.09 0.17 0.86

Thus, our consumption with OLSR running is roughly .9W, which is not terrible and I will talk to Jeremy about getting solar panels to support this. We still need to duty cycle and have a way to scan for APs and to set the association and links appropriately. I will start writing the script today and I'm meeting with Deepak, Jeremy, and Devesh to figure out how this will be packaged.

Thursday, July 5, 2007

more olsrd news

Well, it definitely works now! I'm browsing the web through the two gumstix OLSR nodes I set up. I still think that the D-Link card was flaky and right now I'm using the Linksys WCF-11 Card. For the 2.6.18 kernel a small patch has to be added to linux/drivers/net/wireless/hostap/hostap_cs.c. Essentially, add PCMCIA_DEVICE_MANF_CARD(0x0274, 0x3301) to static struct pcmcia_device_id hostap_cs_ids[]. I created a patch to do it for that particular kernel if anyone's interested, but like I said, it's very easy.

Now, I have multiple topics to choose from for possible directions:
1. I could start writing the "brain" script in something like perl or as a shell script
2. I could start working on duty cycling with tiny nodes

I think I'm gonna get some hot chocolate first to celebrate and then talk with Deepak about the direction.

Last thing, to reduce packet loss I used the firmware TX rate control. You can set it with iwpriv iface txratectrl 1 (0 is to use driver's tx control). You can also find more information about the driver online: http://hostap.epitest.fi/cgi-bin/viewcvs.cgi/hostap/Attic/README.prism2?rev=HEAD

Tuesday, July 3, 2007

let there be mesh

Finally got mesh networking to work, if only for a short while. The configuration I setup was as follows: node A was a verdex board with a netstix and a usb hub connected to it. I put in a D-Link CF card and a D-Link usb card. Node B was a connex board with a CF Linksys card and connected to a router with internet access through ethernet. The last piece in the setup was my laptop which connected to node A. Both of the CF cards acted as access points, with different ESSIDs. The usb card acted as a client connecting node A to B. I ran OLSR on both of the nodes and successfully telnetted to (umass page) and retrieved index.html, which apparently did not exist, so I retrieved a 404 message :). The connecting was done of my laptop. To configure the routes OLSR took maybe 2-3 seconds. I had to specify the AP as a default gateway for all traffic on the laptop, but that seems a natural step either way. The subnet configuration was as follows: Laptop and node A AP were on, node A to node B on, and node B to the router on The router ran a dhcp server. The rest of IPs were configured manually but a DHCP server could be setup with each access point.

This setup worked until I called Deepak to check it out, of course that's when it stopped working. The moment I would try to reach the D-Link access point, I would be disassociated from the AP. I will go back and try to figure it out tomorrow. Overall though, OLSR was very easy to set up, and we should've used it to begin with.

Monday, July 2, 2007

OLSRd part 2

Setting up OLSR turned out to be much easier than click/roofnet. The configuration file is very straightforward and it seems we can handle broadcasting internet access easily by using simple shell scripts (much like the brain script). The current configuration for the verdex works rather well: netstix with wireless for hostap and a usb hub with a dlink card to be used with wlan-ng to connect to other access points. I'm hoping to test OLSRd tomorrow and get some power consumption measurements under various modes. Hopefully, we gain something this way.

Friday, June 29, 2007


Since getting an Atheros card on the gumstix appears to be impossible, I've been looking at other possibilities to get mesh going. Mark pointed out that if necessary we can have 2 cards in the gumstix and we can have a usb hub (so we still get storage). The new approach is to use OLSRd on the gumstix for the mesh and write a few scripts/programs to scan for APs, associate with an AP if it connects to the internet and advertise that to everyone else or connect to a mesh node AP if there are no other options. All of this can be done with shell scripts or perl (or we can do it with Ruby like the Merakis, but perl seems easier).

Today I'm putting together everything necessary and hopefully gonna get a chance to try it out. I need hostapd (to run an AP) and olsrd (to run mesh). We're gonna stick with the prism2 cards and the hostap drivers, since if we can have an interface running an AP and an interface running in Managed mode, out problem is resolved.

A useful guide I found to OLSR: http://tldp.org/HOWTO/OLSR-IPv6-HOWTO/olsrlinux.html

Thursday, June 28, 2007

end of roofnet on gumstix

There is no way to get an atheros card on the gumstix. The CF bus is 8/16 bit wide and the atheros cards are all 32-bit Cardbus. The other option was to use the usb port on the verdex, but that won't work since the only available driver is NDISwrapper, which does not support any modes, but managed and ad-hoc.

The last possible solution is to put two gumstix together connected by an ethernet x-over cable and forward packets from the AP to the mesh node. The power consumption that way would be less than the meraki and we have direct control of what is turned on and can add storage to either one or both gumstix. I think it may actually be the best option at the time.

Monday, June 25, 2007

The verdex board

I tried to follow the same route to get roofnet to work on the verdex, but modifying the patches to work with linux 2.6.19 on the verdex has proven to be quite difficult. I managed to get the kernel to compile, but then errors would appear during compiling of various packages. The one I could not get around was compiling ncurses 5.5. I also tried various revisions to see if older packages would compile fine, but no such luck.

Then I took a slightly different approach and tried to modify the patches for roofnet. In the end I was able to compile a working kernel, but it did not support pcmcia. Now I'm trying to compile a revision known to support CF/PCMCIA with the click patches. I have to repatch the kernel, this time modifying the 2.6.16 patches.

Tuesday, June 19, 2007

roofnet is running

Finally, I was able to get roofnet to run. Editing the shell script to use the iwconfig/prism2_param was fairly straightforward. I ended up putting the card in monitor mode type 2, which resolved the issue of not being able to receive packets. Currently, when click is run, the interfaces are created and it appears that everything is set up. Next step is to test roofnet on two gumstix. I created a rootfs image to drop on them, which should make it go smoothly. I will test this tomorrow. Hopefully it will work. I would also like to have tcpdump on the stix. I'll throw that on tomorrow to see what packets are transmitted.

The only output from click currently is the notice that interfaces went into promiscuous mode and the cryptic "expensive Packet::push; have 4 wanted 32". Not sure what it refers to. Appears to be a statement about transmitting a packet.


Yesterday I picked up TinyOS and spent most of the day getting the environment running. Today I started working with NesC and experimenting with tutorial code. I got radio communication running between telosb motes. I started working with with a set of tinynodes to try and establish radio communication, but have run into a problem where the older v1.1 node will send and receive packets, but the v1.2.1 nodes will not receive packets. So far I haven't been able to determine the difference between the node revisions, and don't know why Rx fails. Specifically, the receive event of the two interfaces I tried (Receive and Snooper) fails to fire, even though both active nodes succeed in sending packets out.

Saturday, June 16, 2007

patched kernel

I managed to compile the 2.6.19 kernel for the gumstix and apply the click patch. I ended up going manually through a number of gumstix patches for the 2.6.18 kernel and throwing out the hunks that were already patched. I then tried to run the config file and pipe the output to click. That failed miserably as it relies on the "wlanconfig" tool from madwifi to create all the devices. I tried compiling madwifi tools and putting them on gumstix, but to no avail, as wlanconfig did not work with the hostap driver. I will try to read the config script next and do all the steps with the hostap configuration tools (or wireless tools).

Friday, June 15, 2007

gumstix kernel and click

In order to get click to run on the gumstix I realized I need to patch the gumstix kernel. Forcing gumstix to use a particular kernel does not work very well, as gumstix come only with patches for particular versions. To force a version one could edit the file target/device/Gumstix/basix-connex/linux.mk.

The hostap driver does actually work with the card we would like to use (WCF12), but our card was busted. Everything appears to be working with the host ap driver (wireless tools shows it and it created an ap interface and a regular one). I'll try to figure out what, if any kernel will work.

Wednesday, June 13, 2007

wireless at last

Once I figured out how to compile the driver (see the edit in the previous post), things went quite a bit smoother. I created a new image of the root file system that includes the modules, threw it on gumstix, loaded the modules with modprobe and was able to associate with an AP and ping google (the ultimate test of every network connection :P). In the process I also ended up setting up NFS which was very easy on gumstix. I followed the gumstix wiki about that: http://docwiki.gumstix.org/NFS, but one thing I would suggest adding to the mount is the soft option, otherwise gumstix may lock up when there is a file request and a timeout occurs. Tomorrow I'll try to throw in the host_ap drivers and see how well roofnet/click will run on the device. After that lots of roofnet code hacking. Not looking forward to it: seems a bit daunting.

Tuesday, June 12, 2007

wlan-ng and wcf11/12

Well, apparently cross compiling wlan-ng for arm is not as simple as it looks. For one the configure script does not prompt for the option altogether and one can manually add it in. The trick is that apparently when making that option is ignored. After looking up errors on google I ended up compiling the damn thing with:

make ARCH=arm CFLAGS+=$(call cc-option,-mapcs-32,$(call cc-option,-mabi=apcs-gnu),) $(call cc-option,-mno-thumb-interwork,) CC=/home/timur/C++/gumstix-buildroot/build_arm_nofpu/staging_dir/bin/arm-linux-gcc LD=/home/timur/C++/gumstix-buildroot/build_arm_nofpu/staging_dir/bin/arm-linux-ld

What should be used and does not require other adjustments is:
make ARCH=arm CROSS_COMPILE=arm-linux-

This does not result in any errors, assuming kernel source and the cross compiler are available. If only it were mentioned somewhere :)
Compile with the same cross compiler as the kernel, otherwise strange errors may appear when loading the modules

Monday, June 11, 2007

gumstix and roofnet

Today we connected to the gumstix over the serial cable and figured out how to flash the device and setup a new root file system. We have compiled roofnet/click for gumstix, configured busybox to provide the necessary tools for future work and setup the image over the serial line. We have also set the wired network interface to be available on a static IP address, which should not cause any issues in the future.

Friday, June 8, 2007

fixed a small bug and found another

In the CVS version from fs.net there is a small bug in the version 1.14 of the library in the str2file function call. Essentially, the parameter to the mydirname function call was incorrect. Here is the diff file from it:
Index: str2file.C
RCS file: /cvs/sfs1/async/str2file.C,v
retrieving revision 1.14
diff -b -u -r1.14 str2file.C
--- str2file.C 16 Dec 2006 03:44:33 -0000 1.14
+++ str2file.C 8 Jun 2007 18:49:16 -0000
@@ -105,7 +105,7 @@
if (fd <>
return false;

- str dir = mydirname (s);
+ str dir = mydirname (file);
int dirfd = open (dir, O_RDONLY, 0);
if (dirfd <>
warn ("%s: %m\n", dir.cstr ());

I submitted the patch to the mailing list and hopefully that will be patched soon.
The other bug is in the sfsauthd daemon. When trying to register a key it quits with an EOF error. I have not been able to track down where exactly the error occurs, but hopefully it's another typo.

In the future I will be working with sensor networks rather than file systems but hopefully I will come back to sfs in the fall.

Thursday, June 7, 2007

sfs adventure

Today I worked on setting up SFS. I got the client installed from the debian testing tree on my laptop and verified it works correctly.

Compiling the server side has been more difficult. The sfskey does not seem to be running correctly as it manages to generate a key but then it does not save it. I will try looking at the source code to find the issue, which I think would be pretty obvious, as the error supplied is "file not found." It seems that there is a small bug where they may be opening a wrong file(?). I hope to have this done tomorrow.

I also subscribed to the sfs mailing list and asked whether cvsweb could be fixed as it'd be much easier to browse the repository, but that will not be a big deal.

Goals for tomorrow:
1. get sfs running fine on the server
2. find the proxy re-signature version of the library
3. begin working on integrating it with the existing code.

Wednesday, June 6, 2007

Day 1

Today I read a paper on another way of implementing access sharing: Ciphertext-Policy Attribute-Based Encryption. The paper outlined how access trees could be used to implement user access control to content. The idea is that each user possesses a number of attributes which a publisher of the content can use to create an access list to allow only users whose attributes satisfy the tree. The tree can include gates at nodes, such as AND, OR, etc, to allow more flexibility. The scheme has the same drawback as WISPR as there is no flexible way of revocation, aside from temporally bound access trees.

I also read the paper describing the file system I will be working with: Separating key management from file system security. I began to look through the documentation available about the filesystem (SFS) in order to begin setting it up tomorrow. In order to set it up I will need to build it on cusp and then try to look into including the proxy re-signature portion back into it.