Brick Buddy 2 Script Storage

It has been a week and would I hope I got a lot done.  Well not so much.  Got side tracked by tasks on other things.  What I have been working on here is integrating the scripting language.  The PIC32MX470F512 that I am using does not have any EEPROM data storage.  So I have been testing different methods for storing the program script in Program FLASH.

While this should be straight forward, it was not for me.  Looking at the forums, several other people got tangled up in virtual memory, physical memory, cached and non-cached.  I wanted something simple.  In the Brick Buddy, I had written a simple interface to the Data EEPROM.  The cell endurance was a minimum of 100K erase/writes.  Given the average usage, I did not see this as a concern.  In this PIC32, the FLASH cell endurance is 20K.  A lot, but low enough to have to consider it.  Also I could erase and write individual bytes previously, in the PIC32 FLASH the minimum erase is 4096 and the minimum write is 512.  I have decided to manipulate and run the script out of RAM, there is lots of this.  The new script storage process, from a high level, looks like this:

• Read the stored script from FLASH into a buffer in RAM
• Modify, Add, Delete and run from the buffer
• Store the script in FLASH only on command, either USB or Bluetooth

The script will always be on the PC or the Bluetooth device (a change that is now needed).  Unless someone is extreme on the saving, 20K cycles will last 3- 5 years ( 5 years-> 4000/year or more than 10 a day).  I can also alocate a second 4096 block of Program FLASH and double the life.  There will also be lots of Program FLASH when I am done.  One big upside with 4096 as the minimum erase, is that means there is space for over 1000 instructions.  Now I need space for other variables, but I can see setting the limit at 1000.

I took one of the Harmony examples, flash_modify, and did some major surgery by manually merging it into a project that had a system console and system debug.  The biggest issue was dealing with the 512 byte write size, the flash addresses being in bytes and the script buffer being in unsigned longs (32 bit).  The final issue was the virtual vs physical addresses and cached vs non-cached.  I ended up using 0xBD070000 as the address and that works. Once I finish testing tomorrow,  I will post the project for everyone to use on the MyMakerTools website.  Look down the page for Other Goodies.  

UPDATE:There was much discussion in the forums about using KSEG1 (non-cached) addresses instead of  KSEG0 (cached) addresses.  I tested it both ways and this code seems to work either way.  Tracing through shows the low level PLIB routines use the compiler macro, KVA_TO_PA to ensure that the correct addresses are used.

Here is the basic state machine.

/**********************************************************
 * FLASH NVM tasks routine. This function implements the
 * FLASH NVM state machine.
 ***********************************************************/
void APP_NVM_Tasks ( void )
{
   unsigned int x;
   uint32_t  bufOffset, addrOffset;
  
   x = APP_DATABUFF_SIZE;
   switch(appData.nvmState)
   {
      case APP_STATE_NVM_INIT:
        appData.nvmHandle = DRV_FLASH_Open(DRV_FLASH_INDEX_0, intent);
        appData.nvmState = APP_STATE_NVM_FILL_DATABUF_AND_ERASE_STATE;
        appData.nvmRowCount = APP_DEVICE_ROW_COUNT;
        PrintBuffer(true);
        break;

      case APP_STATE_NVM_FILL_DATABUF_AND_ERASE_STATE:
        for (x = 0; x < APP_DATABUFF_SIZE; x++)
        {
            scriptBuffer[x] = x;
        }
        LED_BLOff();
        LED_YLOff();

        PrintBuffer(false);
        /* Erase the page which consist of the row to be written */
        DRV_FLASH_ErasePage(appData.nvmHandle, APP_PROGRAM_FLASH_BASE_ADDRESS);
        appData.nvmCurrentRow = 0;              // set row value to zero
        appData.nvmState = APP_STATE_NVM_ERASE_COMPLETION_CHECK;
        break;

      case APP_STATE_NVM_ERASE_COMPLETION_CHECK:
        if(!DRV_FLASH_IsBusy(appData.nvmHandle))
        {
            PrintBuffer(true);
            appData.nvmState = APP_STATE_NVM_WRITE_START;
        }
        break;

      case APP_STATE_NVM_WRITE_START:
        /* Erase Success */
        /* Write a row of data to PROGRAM_FLASH_BASE_ADDRESS, using databuff array as the source */
        //PrintBuffer(true);
        bufOffset = appData.nvmCurrentRow * APP_DEVICE_ROW_SIZE_DIVIDED_BY_4;
        addrOffset = appData.nvmCurrentRow * APP_DEVICE_ROW_SIZE;
        DRV_FLASH_WriteRow(appData.nvmHandle, APP_PROGRAM_FLASH_BASE_ADDRESS + addrOffset, &scriptBuffer[bufOffset]);
        appData.nvmCurrentRow++;
        appData.nvmState = APP_STATE_NVM_WRITE_COMPLETION_CHECK_AND_VERIFY_CHECK;
        break;

      case APP_STATE_NVM_WRITE_COMPLETION_CHECK_AND_VERIFY_CHECK:
        if(!DRV_FLASH_IsBusy(appData.nvmHandle))
        {
            PrintBuffer(true);
            /* Verify that data written to flash memory is valid (databuff array read from kseg1) */
            if (!memcmp(&scriptBuffer[bufOffset], (void *)(APP_PROGRAM_FLASH_BASE_ADDRESS_VALUE + addrOffset), DRV_FLASH_ROW_SIZE))
            {
                appData.nvmState = APP_STATE_NVM_SUCCESS_STATE;
            }
            else
            {
                appData.nvmState = APP_STATE_NVM_ERROR_STATE;
            }
        }
        break;
        
      case APP_STATE_NVM_ERROR_STATE:
        /*stay here, nvm had a failure*/
        LED_BLOff();
        LED_YLOn();
        if (appData.nvmCurrentRow < APP_DEVICE_ROW_COUNT)
        {
            appData.nvmState = APP_STATE_NVM_WRITE_START;
        }
        else
        {
            PrintBuffer(true);
            appData.nvmState = APP_STATE_NVM_IDLE_STATE;
        }
        break;

      case APP_STATE_NVM_SUCCESS_STATE:
        LED_BLOn();
        LED_YLOff();
        if (appData.nvmCurrentRow < APP_DEVICE_ROW_COUNT)
        {
            appData.nvmState = APP_STATE_NVM_WRITE_START;
        }
        else
        {
            PrintBuffer(true);
            appData.nvmState = APP_STATE_NVM_IDLE_STATE;
        }
        break;

      case APP_STATE_NVM_IDLE_STATE:
        LED_BLToggle();
        LED_YLOff();
        break;
   }
}

Brick Buddy 2 Serial Ports -- DONE

Well this took way too long.  Though I will say that on Friday and Saturday I was working on this while attending Bricks by The Bay.  As in the last post on serial integration, I need to contemplate the process of testing this.  After some consideration, I decided that mounting the BT Module was not the correct approach.  I would never know if it was the serial port or the BT Module causing the problem.  And the issues I had, verified this approach.  My goal was to have an interrupt driven process for both Tx and Rx.

What I did to test:

1. I used a connector intended to reprogram the Bluetooth Module (tag-connect) as the interface to the second serial port (USART 1).
2. The first serial has not changed, STATIC, BLOCKING.
   
3. The second serial port was implemented as STATIC, INTERRUPT, CALLBACKS and should be NON_BLOCKING.
4. Used the Harmony generated Interrupt routines, but disabled the ERROR checking routine.
5. Developed the CALLBACK functions for Rx and Tx.
6. Developed ring buffers for both Tx and Rx.
7. Developed test code that filled the Rx buffer with a string, that then transfers this to the Tx buffer and begins the transmit.
8. Finally  I had test board with two FTDI USB to Serial converts (from Sparkfun) to observe the two serial ports in action, see below.  Each serial port is connected to a Tera Term application on the PC.
   

I created the call back routines and registered them.  Both the Tx and Rx callbacks use the STATIC driver API, DRV_USART_WriteByte and DRV_USART_ReadByte to move a single byte into or out of the USART and into the Tx or Rx buffers.  In both cases, the callback function attempts to either fill the USART Tx Buffer (for Tx) and read all of the bytes in the USART Rx buffer (for Rx) before exiting the callback function.  This reduces the chances of over runs.  Finally in the initialization of the APP state machine, I included a small piece of code to load the Rx buffer with a character string of 10 characters. In the work portion of the APP state machine,  any time there are bytes in the Rx buffer, they are transferred to the Tx buffer.  In theory typing into the Tera Term applicaion should send bytes to USART 1, which in turn will send the bytes right back out to the Tera Term application.

Here is where the problems started.  I had seen a Microchip Forum thread ( see the last entry) that had discussed how one person did it. But they had disabled the Tx interrupt for reasons not stated as well as the ERROR checking interrupt.  I was getting inconsistent results.  The first byte would go through, but subsequent key presses would get into the Rx buffer, but would never be sent out.  The only good thing was that it appeared that the Rx path was working as expected.

To prove that the Tx path was working I added a section like this, using bare metal code

while (true)

{

    while U1STAbits.TRMT == 0);

    UITXREG = 0x55;

}

This worked as expected with the Tera Term application filling up with capital Us.  At least I knew there was a path from the PIC32 to the FTDI part.  I went back to a more traditional approach of using this USART API call after checking the transmit holding register was empty, DRV_USART1_WriteByte, then updating all of the variables.  But again weird behavior.  After placing printF type statements everywhere and tracing through, the above API looks like this:

void DRV_USART0_WriteByte(const uint8_t byte)
{
    /* Send one byte */
    PLIB_USART_TransmitterByteSend(USART_ID_1, byte);
    SYS_INT_SourceEnable(INT_SOURCE_USART_1_TRANSMIT);
}

What I finally figured out was happening, involves the second statement. The interrupt was being enabled and the Tx callback function was called before any of the variables in the main routine could be updated.  The Tx callback function is essentially the same routine and updates the same variables.  So I modified the APP state machine code to follow this process

• clear the Tx interrupt flag in the USART
• disable the Tx interrupt for the USART
• use the PLIB to send one byte  // the API will enable the interrupt before I am ready
• update all the buffer variables
• enable the Tx interrupt for the USART

Now it works like I want it to.  Loading the first byte into the Tx register of the USART, kicks off the transmit process.  As long as there are bytes in the Tx buffer, the Tx interrupt callback will handle them and the main application can do whatever it needs to do.

Once I cleanup the code a little more and finish testing , I will update this post with the entire project I used to get these two serial ports working.

*******************************************************************

Here is the link to the Harmony Project.  The Zip file is towards the end of the page

PIC32Fixes-DualUSART.zip

*******************************************************************