A few comments on hash-codes

Apropos of my comments on databases, there is something that any extensive database must have, and that is a "guaranteed" unique key associated with the root entry for any set of related data - a unique account number, if you like.

There are, of course, ways and means to achieve something that approximates this and most of those means use things like MD5 hashes or Global Unique Identifier Numbers.

The latter is a 128-bit number expressed as 32 hexadecimal digits. They are generated with a small but finite chance of collision, but for most purposes, they are effectively unique.

But, the question, is how does one go about generating these numbers?

The answer is simply - complicated mathematical functions.

Now, for the purpose of having a unique identifier that is automatically generated and that will be associated with the root entry for each accession in our Database of Curiosities, and which will remain, no matter what else we change at a later date, it is possible to come up with a relatively simple way of hashing some kind of data (called a message digest).

In this case, the entire primary entry is combined with a date and time string and the (supposedly unique) accession number, and then mangled in order to produce a 128-bit code.

It doesn't matter that the code is meaningless to your eyes, it is a reference number that should be effectively unique in the world. If you then prefix that code with a hexadecimal representation of the accession number, then you have an excellent chance that there will be no collisions anywhere or at any time.

Since we are not using this method to generate data integrity checksums, nor are we using it to encrypt data, then there is no need for the complexity of the MD5 or GUID generation code.

I constructed this function in Excel for the purpose of testing, and so here is the code:

(apologies for the long lines)

' Simple hashing function to generate a unique record identifier which will not collide with other
' record identifiers.

Public Function HashRecord(strIn As String, strTime As String, strID As String) As String
    ' strIn -       The data string to be hashed.
    ' strTime -     A string representation of the date and time of the function call.
    ' strID -       A string representation of the accession number
   
    Dim numHash(18) As Integer
   
    strFeed = strTime & strID & strIn & strID & strTime
   
    For i = 0 To 18         ' explicitly zero the working array.
        numHash(i) = 0
    Next i
   
    byCarry = 0
   
    For i = Len(strFeed) To 1 Step -1
        strNext = Mid(strFeed, i, 1) ' extract the next character
        chrVal = Asc(strNext)        ' convert the character into an 8-bit ASCII value
       
        numHash(1) = ((numHash(1) * 8) + byCarry) Xor chrVal
            ' shift the binary bits of the number up 3 bits, add whatever
            ' carry value came from the last calculation and XOR with the
            ' next byte of the string to be hashed.
           
        byCarry = 0 ' the carry value
       
        If numHash(1) > 255 Then    ' calculate the carry, and the value of the first byte
            byCarry = Int(numHash(1) / 256)
            numHash(1) = numHash(1) And 255
        End If
       
        For j = 2 To 16             ' ripple the change up the entire array
            numHash(j) = (numHash(j) * 8) + byCarry
                byCarry = 0
                If numHash(j) > 255 Then
                    byCarry = Int(numHash(j) / 256)
                    numHash(j) = numHash(j) And 255
                End If
        Next j

    Next i

     If byCarry Then    ' if there is still a carry, then wrap it around and add it into the first byte.
         For j = 1 To 16
            numHash(j) = numHash(j) + byCarry
                byCarry = 0
                If numHash(j) > 255 Then
                    byCarry = Int(numHash(j) / 256)
                    numHash(j) = numHash1 And 255
                End If
        Next j
    End If
   
    hexHash = ""
   
    For j = 1 To 16 ' convert to hexadecimal (with leading zeros)
        hexHash = hexHash & Right("00" & Hex(numHash(j)), 2)
    Next j
    hexHash = Left(LCase(hexHash) & "0000000000000000000000000000000000000000000000000000", 32) 
' fix the length, and make the letters lower case
    hexHashX = Left(hexHash, 8) & "-" & Mid(hexHash, 9, 8) & ":" & Mid(hexHash, 17, 8) & "-" & Mid(hexHash, 26, 99) ' break up the string of digits
   
    hexHash = ""
    For i = 1 To Len(strID) ' convert the accession number to hexadecimal
        hexHash = hexHash & Right("00" & Hex(Asc(Mid(strID, i, 1))), 2)
    Next i
   
    hexHash = Right("0000000000000000" & hexHash, 16) ' just 8 characters to be used

    HashRecord = Left(hexHash, 8) & "-" & Right(hexHash, 8) & ":" & hexHashX
   
End Function



.
The output looks something like this:


00004D30-30303030:66b531da-92377f48:7d279dcd-6e7d107
The first two groups are the accession number (M00000), the remaining four groups are the digest.


Of  course, it is quite possible that I will simply end up using something much simpler.

Field Mate part 3 - Digital Humidity and Temperature

As a part of this project, I wanted to add in an environmental hygrometer as part of the environmental sensorium, and ordered a DHT-11.

The device is itself a hybrid, containing a chip, a capacitive humidity sensor and a thermistor. It is actually cheaper to order these things attached to a small circuit board (breakout board) intended for hobbyists - especially as they include the recommended resistor and capacitor, and are thus ready for plug in and go.

While the device sports four pins, only three of them are in use.

Unlike most devices which produce an analog output, or those that are accessed using I2C or SPI data buses, these little boxes of tricks have a single-wire serial interface that is, frankly, difficult to use.

The Internet maker community to the rescue! I was originally going to use a PIC microcontroller to turn the DHT-11 into an I2C device, but I found that there is a library available for the Arduino (since that is what I am currently using).

A few false starts, and I have access to the world of Relative Humidity and Temperature measurement. From that, it is possible to calculate Dew Point (the temperature at which that atmospheric moisture would begin to condense) and Heat Index (the apparent temperature of the atmosphere). Neither of these numbers are easy to convert using a calculator although they only take a few lines of code in a program.

Before I deal with the software, there are a couple of caveats regarding the DHT series of sensors -

The capacitive humidity sensor absorbs moisture rapidly, but will give it up fairly slowly, so breathing into the device, getting it wet or taking it into a steamy kitchen is going to have an effect that lasts a couple of hours.

The DHT-11 is integer only (which I didn't realise), but the data capture library is compatible with the whole of the DHT range (DHT-11, -21, -22, -33 & -44), which means that they are plug compatible with both software and hardware. Upgrade to the higher precision devices is therefore quite simple.

The DHT-xx series are rather slow devices, and so should not be interrogated too frequently - this is taken care of quite naturally by the device library, as it delays other program activity until the data has been read (blocking device).

 

The library and more information is available from the Arduino Playground -

http://playground.arduino.cc/Main/DHTLib 

 The latest edition of the library is available from Rob Tillaart on GitHub -

https://github.com/RobTillaart/Arduino/tree/master/libraries/DHTlib

The source code for the wrapper which I am using, based on Rob's examples and other contributions, follows:

// dhtwrap.h
//
// ####################################################################
// ####################################################################
// ##
// ##  dhtwrap.ino (dhtwrap.h)
// ##  version 1.00.00
// ##  date    12-Aug-2015
// ##  author  Alysson Rowan (AlyssonR)
// ##   BASED on the work of Ron Tillaart
// ## 
// ##   alyssonrowan @ gmail.com
// ##
// ####################################################################
// ####################################################################
// ##
// ##  This program is provided on an as-is basis without warranty.
// ##  No claim is made as toward operability or fitness for purpose
// ##  whatsoever. No liability can be accepted for any loss or damages
// ##  howsoever caused in respect of this software.
// ##
// ##  This software is made freely available under the terms of the
// ##  GNU General Public License V2. In short, you are allowed to do
// ##  anything with this program except sell it and hide the
// ##  source code.
// ##
// ##  In addition, this program is "postcard ware" – if you find it
// ##  useful then please send me an e-mail and tell me about your
// ##  application.
// ##
// ##  Your comments and suggestions are much appreciated.
// ##
// ####################################################################
// ####################################################################

// ####################################################################
// ####################################################################
// ####################################################################
// ####################################################################
// ##
// ##  DEPENDENCIES:
// ##  -------------
// ##
// ## DHTlibrary by Rob Tillaart -
// ## Latest version is available at:
// ##                   https://github.com/RobTillaart/Arduino
// ##
// ## Library files at:
// ##                   https://github.com/RobTillaart/Arduino/tree/master/libraries/DHTlib
// ##
// ## File tested with DHT library version 0.1.20




#include
#define DHTLIB_ERROR_DEVICE     -9  // - additional error code

dht DHT;

double DHTtemperature;  // Return values from DHTxx device.
double DHThumidity;
double DHTdewpoint;
double DHTheatindex;
int DHTerror;

void DHTGet(int DHTmodel, int DHTpin ) {
// Exit error codes:
// DHTerror =    0 :  No error
// DHTerror =   -1 :  Checksum error
// DHTerror =   -2 :  Timeout
// DHTerror =   -3 :  Connection error
// DHTerror =   -4 :  ACK-L error
// DHTerror =   -5 :  ACK-H error
// DHTerror =   -9 :  Invalid device selection
// DHTerror = -999 :  Unspecified Error

  int chk;
  switch (DHTmodel) { // Get data from the DHTxx module - using the appropriate read function.
 
    case 11:
                chk = DHT.read11(DHTpin);
                break;
    case 21:
                chk = DHT.read21(DHTpin);
                break;
    case 22:
                chk = DHT.read11(DHTpin);
                break;
    case 33:
                chk = DHT.read33(DHTpin);
                break;
    case 44:
                chk = DHT.read44(DHTpin);
                break;
    default:
                chk = DHTLIB_ERROR_DEVICE;
                break;
  }           
 
  DHTtemperature = -999.0; // Set default (ERROR values, rather than end up with NaN
  DHThumidity    = -999.0;
  DHTdewpoint    = -999.0;
  DHTheatindex   = -999.0;

  switch (chk) { // Error test and set the return data

    case DHTLIB_OK: 
                DHTtemperature = DHT.temperature;
                DHThumidity = DHT.humidity;
                DHTdewpoint = dewPoint(DHT.temperature, DHT.humidity);
                DHTheatindex = heatIndex(DHT.temperature, DHT.humidity);
                DHTerror = DHTLIB_OK;
        break;
    case DHTLIB_ERROR_CHECKSUM:
                DHTerror = DHTLIB_ERROR_CHECKSUM;   
        break;
    case DHTLIB_ERROR_TIMEOUT:
                DHTerror = DHTLIB_ERROR_TIMEOUT;
        break;
    case DHTLIB_ERROR_DEVICE:
                DHTerror = DHTLIB_ERROR_DEVICE;
                break;   
    default:
               DHTerror = chk;
        break;
  }
}


// dewPoint function NOAA
// reference (1) : http://wahiduddin.net/calc/density_algorithms.htm
// reference (2) : http://www.colorado.edu/geography/weather_station/Geog_site/about.htm
// Thanks to Rob Tillaart who came up with this.

double dewPoint(double celsius, double humidity)
{
    // (1) Saturation Vapor Pressure = ESGG(T)
    double RATIO = 373.15 / (273.15 + celsius);
    double RHS = -7.90298 * (RATIO - 1);
    RHS += 5.02808 * log10(RATIO);
    RHS += -1.3816e-7 * (pow(10, (11.344 * (1 - 1/RATIO ))) - 1) ;
    RHS += 8.1328e-3 * (pow(10, (-3.49149 * (RATIO - 1))) - 1) ;
    RHS += log10(1013.246);

        // factor -3 is to adjust units - Vapor Pressure SVP * humidity
    double VP = pow(10, RHS - 3) * humidity;

        // (2) DEWPOINT = F(Vapor Pressure)
    double T = log(VP/0.61078);   // temp var
    return (241.88 * T) / (17.558 - T);
}



// heatIndex function NOAA
// reference (1) : http://www.colorado.edu/geography/weather_station/Geog_site/about.htm
// Thanks to Rob Tillaart who came up with this faster version.

double heatIndex(double celsius, double humidity)
{
  double farenheit = (celsius * 1.8) + 32;
  double farenheitsq = farenheit * farenheit;
  double humiditysq = humidity * humidity;
  double heatindex = celsius;
  if ((farenheit >= 80) and (humidity >= 40)) {
    //Heat index is only valid for temperatures 80F upwards AND for R.H. 40% upwards
   
    double c1 = -42.379, c2 = 2.049015, c3 = 10.14333, c4 = -0.224755, c5 = -6.83783e-3 ;
    double c6 = -5.481717e-2, c7 = 1.22874e-3, c8 = 8.5282e-4, c9 = -1.99e-6 ;

    double A = (( c5 * farenheit) + c2) * farenheit + c1;
    double B = (((c7 * farenheit) + c4) * farenheit + c3) * humidity;
    double C = (((c9 * farenheit) + c8) * farenheit + c6) * humidity * humidity;
    heatindex = A + B + C;
   
    heatindex = (heatindex - 32) /1.8; // We are using Celsius throughout.
  }
  return heatindex;
} 

While it is usually considered most improper, I tend to use global variables for communicating blocks of related data between parts of the program. This makes things simpler for beginning users. I could have used a structured variable, but that makes the wrapper more complicated for a novice user. By calling

DHTGet(deviceSuffix, DigitalPinNumber);

then the temperature, relative humidity, dew point and heat index will be placed in  the variables

DHTtemperature, DHThumidity, DHTdewpoint and DHTheatindex respectively. The error status will be in DHTerror, and will be zero if there is no error.

All temperatures are expressed in degrees Celsius, the relative humidity is expressed as a percentage.

Field Mate Part 2 - Open Your Eyes to an Open Sensorium

A few posts past, I addressed the idea of Field Buddy Field Mate - a portable device for capturing environmental data while in the field. Having had a chance to play with a bit of magnetometry (amongst other toys), and finding that I need to build an intelligent thermostat, I have come to the conclusion that there is something lacking in the world of hobby/amateur electronics - and that is a properly integrated set of devices and software for microcontroller use.

I have decided, therefore, that it is time to formalise the Field Mate concept into a more general project.

To this end, I have decided to commence work on what I am calling The Open Sensorium Project.

The aim is to produce a series of software and hardware modules that can be built up and put together in order to build custom instrumentation. Based around Data Capture using sensors (i.e. a sensorium), the modules will provide both sensing, primary processing and export of data via displays, data streaming and data caching (using SD cards).

It will not only provide an open sensorium, in the sense that the data system's "eyes" are wide open, but will also be an open project - open source software, open source hardware (albeit, using a lot of off-the-shelf modules) and free to use and modify.

Given the relatively low cost of microcontrollers, it would be possible for each major module to be separately intelligent.

I hope that there will be a lot of cross-pollination with various other projects, and between the developers using a range of microcontroller systems including, but not limited to, Microchip's PIC, Parallax Propeller, Atmel AVC / Arduino and Raspberry Pi,

As an open project, the material will be released into the wild with few restrictions, and subject to the Gnu General Public Licence.

The rules will be simple - open source may not form a part of a closed source project unless those portions that are open source remain open source. Derivative works are brilliant. Respect and acknowledge the intellectual rights of those whose work you are building upon - and retain any copyright notices that form a part of the source that you are using.

Finally, there will be standards for various parts of the project - standards for quality of product, for quality of documentation and for communication protocols between modules - and to the outside world.

I hope that others will want to get involved in various ways - even if it is only through an eMail saying that you found it useful.

I look forward to hearing from you. I will pass on the web site address once I have settled upon one. I have set up a project page on Sourceforge at: https://sourceforge.net/p/open-sensorium/



For those who care about these things - the font is Neuropolis, the logos were made in MS Word 2013 and either screen-captured or copied to Inkscape.