Flutter Apps for Scientific data collection

This year I spent some time working at the Royal Botanic Gardens Sydney as part of Maurizio Rossetto’s Ecology Team. While there I was asked to design a data collection app that would work seamlessly with their current internal database system.

The apps requirements were a bit scary! It needed to work on Android and iOS. Save data on the local device. Sync a list of user accounts, plants and tenures to the local device from a web database. There needed to be an admin interface to set up and manage the lists of user accounts, plants and tenures. The interface needed to ensure users filled out all the necessary details and the data was error free. Data entry needed to be easy and not restrict end users. The app needed to work in remote areas without an internet connection. Read barcodes and collect gps details. Once all data was collected it needed to be converted into a bizarre custom format that could be read in by a database held inside the Botanic Gardens. Oh and I needed complete the project by myself in 6 months with zero budget to pay for online databases etc.

Any normal person would probably have said no to the project, but I was up for a challenge and after some background research, felt it was doable. If you’re thinking about trying to do something similar, here’s a rather long run down on how I managed to pull this off and some of the concepts that might help you do the same.

Preparation

I spent 2 years volunteering at the gardens as an undergrad. During this time I absorbed the processes and how they could be improved. I also spent time learning about the infrastructure, the network capabilities, the current database system, how I could work within these frameworks and the people I could get answers from. Even though I was given 6 months to complete the project, I spent a year or more preparing before saying yes. Obviously this isn’t always possibly but take as much time as you can to know the landscape.

Get all business requirements

Ask anyone and everyone involved in the project what is needed and what hoops you will have to jump through to make these things happen. Find out from the word go if it is possible to fulfill all the requirements! Often the people asking you to do the work don’t really understand what they are asking for, so if you are going to fail, then fail as quick as possible!

Is it possible to build the final product

Since the beginning of time, cross platform mobile development has been a difficult process! Luckily as I started looking into this project Google had just released the beta version of Flutter https://flutter.dev/docs/get-started/install. Flutter is probably the future of programming and I don’t think I would have taken on this project if I hadn’t discovered it. You can write code once and compile it for many platforms. Flutter is production ready for Android and iOs and will soon be production ready for web apps. Support for Mac, Windows, Linux(?) desktop apps and embedded platforms are also on the way.

Cross platform… Tick.

To write Flutter (dart) code you need a development environment. VScode is free and has good support for Flutter on mac and windows. For iOS you will also need Apple’s Xcode, again free, which in turn requires you to have a mac.

Development environment… Tick.

Flutter has a good set of libraries.

gps library… Tick.
barcode scanning library… Tick.

Flutter can access Google’s Firebase/Firestore online nosql database, which is free up to a certain point. Luckily the app I was building ‘should’ stay under the free limits.

Online database… Tick.

Flutter has a library called sqflite, (an SQlite database for flutter). This database can be written into your app for local persisting data. You can create and access the sqflite database through code so it’s like using an ORM (Object relational manager) and there’s no need for custom php scripts between the database and interface.

Local offline database… Tick.

Flutter has built in support for JSON. I decided to write data to the internal database as json encoded strings. This meant I could save the objects that the interface used directly to the database. When the app needed data it could read it from the database and it was ready for the app to make use of. This has lots of benefits and sqflite has good functions to help this useage.

To get data off the local device and back to the Gardens internal database I decided to go with a simple email. To set up external access to the Gardens network was almost impossible with IT restrictions etc.

The data was previously collected manually by logging into a website, downloading a spreadsheet then imported it into the Garden’s database. This new app would export spreadsheets in the same format the previous app used and could make use of the current database import scripts. Emailing the data meant that it could go straight to the database administrator and the person who entered the data, allowing instant access to the data.

Flutter has a library for emailing.

Free google email account… Tick.
Export data… Tick.
Everything free (up until now)… Tick
.

I did have to spend money to get a Google and Apple Developer account which allowed me to upload to the Android and Apple app stores.

Make sure you have support

Bob Baxley has a concept of ‘finding the path to yes’. In my experience I would say you need two paths to yes, one up and one down. You don’t want to be in a position of answer to 3 different managers! Find one person upwards who can give you the answers you need, when you need them. If possibly, make it so you respond only to this person. This yes person should trust and defend your decisions but will also be hindered by political or business rules that you’re unaware of. You in turn should be comfortable trusting they will make the best decision for the project.

The same goes with finding one reliable yes down. This is an end user to help you make quick decisions about usability. Don’t discount other people’s feed back but these two key people can be your main sources of ‘truth’ and you really want to freely discuss ideas with these two people.

There will be resistance

Not everyone on your team will support you and you may find yourself wasting a lot of time defending decisions. Try to keep or get naysayers onside, they may have valid feedback and your job is hard enough without stoking their fires. If they do go to the darkside call in your ‘yes up’ person to help you out.

Pick your fights, one way to conserve energy is to say yes to ‘almost’ everything. People who like to be right will often leave you alone once they have their yes, so be agreeable without making promises. When you face a real obstacle you should be more able to change minds having not used up all your ‘cry wolf’ points.

Communicate

Keep in constant contact with all the stakeholders, you don’t want to give or receive any surprises. If like me you’re an introvert, you will find this difficult but it is a vital part of success and if you can’t achieve this you might be lining up for a failure! People will help friends so try to make sincere friendships with your team, you catch more yes’s with honey.

On the flip side don’t assume that everyone else cares about the project as much as you do. It might be on your mind 24/7 but it is only a small part of the other players job. So use other people’s time wisely and respect how they like to work, i.e. book in time in their calendar or take them for a coffee if that’s how they work.

Keep a master list

Write everything you need to do in a master to do list. As you do them you can move them to a ‘done list’ with date if you are keen. This can help you keep track of what needs to be done and make you feel like you are making little wins along the way. Even if you are about to work on a new part, write it in the to do list first, you might get distracted and not do it. Once I start coding I also write this list in the code itself and check it off as I go, This is a good way of tracking changes between versions.

MVP

Plan a ‘Minimal Viable Product’. Work out the minimal requirements for the app to be useable. Remove any fluff and nice to have features (these can be added later), keep it as simple as possible. Your app should be a good foundation for the future with just enough functionality for the job required now. You will find management and end users asking for extra features and you need to confident in deciding if they are truly needed in the first release. It’s better to finish an app that does some of the job well, than to have an unfinished app that tried do everything!

Apple regularly releases new app versions with small incremental changes instead of taking years to release a major update. Someone could bring out a better app tomorrow so try to be the first to bring out the new feature.

UX & Design Thinking

Read about these! Successful apps have good UX (user experience), and user experience development processes can help make your app successful. This can help you understand who will use your app, what they want from it, where they will have problems, what is important to have and not have and above all it will help you solve problems before you start writing code.

Coding is difficult, expensive and time consuming but good design is probably the most difficult part of developing any product. No matter how good the code is, it won’t fix bad design. And no matter how good your visual design is, it is useless if the end users refuse to use your app. You can build Rome, but everyone might decide to go to Paris. When people have a choice they will pick the best experience for them, UX is everything!

Agile, Kinda…

Agile concepts can help, don’t take them as gospel. Aim for small achievable milestones, work quickly towards these, test the results, fix issues, test again, and so on. Do this at all stages, do it early and quickly, find issues before they become expensive to fix! Check out what Bob Baxley says about Apple’s weekly design process. https://www.designbetter.co/podcast/bob-baxley

Consistancy

Make your design look and feel seamless. Use common design patterns so users intuitively understand your app. Make it look and feel like the rest of the companies design – only better!.

Prototype

A lot of development projects fail so prototype carefully before starting any coding! Design low fi, sketch, show end users, refine.

Draw layout ideas, test them with the people who will use the final product. You should already have a good understanding of business and infrastructure related issues so design within these. Iterate the designs until you have a clear understanding of what the user wants and needs. You should be able to do this sort of work with in minutes, even while talking to the end users. Do this for every part of the design, you can have good ideas roughed out and refined with in minutes to hours. If you iterate a few times, you might take a couple of days.

If you can’t come up with a solution, go away, think about it and come back to the end user when you have a new idea. Try your ideas out with as many people as you can. You want to find issues, at this stage issues are your friends. – issues can kill your project further down the track! You are aiming to have a good refined design from the beginning of the app to the end, on paper and in your head.

I start with paper drawings for all ideas. Once I think they are clear, I will draw them in illustrator, Indesign, Photoshop, word, what ever I have access to. Quick is good! Bang it out, if it’s bad you can change it just as quickly! Try to make a complete visual design of the app and document it for the next stage.

Now make a prototype. There are lot’s of dedicated programs for app prototyping but even something as simple as Powerpoint or Keynote can work. I use Filemaker* for this, you can get a free trial version for Mac or PC. The idea is to build what looks like an application with all the buttons and text that the end user would expect. Again start lo fi, simple black and white with text. Make the buttons and interface feel like they work. The next button should go to the next page etc. Make end users try out real world tasks in the prototype and watch how they work in your app. If they are having problems, refine the design and test it again. The idea is to test that users can do everything they need and do it easily. Don’t let them focus on the visual design, that’s next! Try not to waste the users time, give them a good chunk of elements to test but not too much that it’s difficult for you to make changes to. Remember issues are your friends at this stage, so try to find them all.

After you have a good lo fi prototype, up the game and start prototyping the look and feel. Bring in colour and visual design elements. Like everything up until now, this can be incremental and evolve. Most of these elements should have already been tested on paper and be in your documented visual designs so you should have a good idea how they will work out. Still they need to be tested in a working prototype and adjusted based on user feedback. You should keep testing and evolving the design until you have a polished, ‘finished’ looking app that does everything the users require. This is were your ‘yes down’ person can come in handy for testing and feedback.

*Filemaker is pretty amazing and may be enough for your needs. In this case the final product needed to work on iOS and Android so Filemaker, though good enough for developing the prototype, wasn’t a good fit for the final product. It’s also pretty expensive for a science department with zero budget. https://www.filemaker.com/

Don’t let users make mistakes

If users are making mistakes with your design don’t blame them. Your design is flawed. Talk to end users and find out where they are having problems. You are aiming for a simple, logical interface that prevents mistakes. Swallow your pride, it’s hard I know! Remember rule 51. ‘Sometimes you are wrong!’.

Build, Test, Fix, Repeat

Once you’ve done all your user testing and your ‘yes up’ and ‘yes down’ people sign off on the prototype, it’s time to start coding – but before you get too excited just double check everyone is happy! Another 5 minutes now could save you days of coding.

Code away

Flutter/dart was new to me, so I searched the web for examples of code that did all the different things I needed. How do you make a text field? How do you put an image on the page? How do I make a list of check boxes? How do I move from one screen to another? How do I get a gps location? How do I read a barcode… Then I figured out how each of these things worked, distilled them to their basics and put these snippets of code aside till I needed them.

When I started coding, I built the app from the beginning of the user experience, i.e. 

First screen was the spalsh screen, then the log in screen which required user accounts. User accounts needed to be stored in an online database, the app required access to the online database, the app needed to download a the users accounts and store them in the local database. The log in screen probably took a couple of weeks to build but laid the foundations for much of the app to come.

As I built each section I showed it to end users, got their feedback, made any changes needed and then moved on to the next section. Work quickly don’t over engineer, use the simplest process you know. Spend a bit of time investigating the available processes but don’t get paralyzed by over thinking. There are always going to be better ways of doing what you are trying to achieve but in the end the user won’t care what the code looks like!

Try to make your code modular and reusable but again it is best just to get something finished and clean it up later. Having said this, you rarely get the chance to go back and do the work again so do the best job you can as quickly as you can. Writing good comments throughout the code will be life saving when you come back to it later. Think about other developers having to work on your code in the future.

With rapid development environments like Flutter you are constantly having to decide if it is better to just bang out duplicate code or spend time to write it cleanly in one place. Making a good module (class/function/widget) that you know will be used over and over can save you a lot of time down the track, and I try to separate most code into reusable chunks. If I come across an issue that affects multiple parts of the app I will then spend the time to move that code to a seperate module and reference it in the rest of the app.

This process took about 3 months.

You will have to change parts of your code

Originally I used the online Firestore database for both online and offline. The Flutter Firestore library automatically stores data on the local device and when you have an internet connection it will sync your data with the online database. This seemed like a good idea at first but quickly became an issue. Because it synced every data change made, it was going over the limit for free daily useage. Automatic syncing also confused the end user. They weren’t sure if or when the data had saved to the cloud. I decided to make a definite divide between the mobile database and web database by changing the local database from Firestore to sqflite. This solved both problems but added a couple of weeks to the development time.

Another problem that came up during testing was a flaw in my data structure. I original designed the data to be store in the following objects. Sites have Populations. Populations have Collections. Unfortunately I figured out that I needed to add in a new object of Individual. So it became Sites have Populations. Populations have Individuals. Individuals have Collections. This wasn’t a huge issue but added another week to the development time, which could have been saved if I had figured it out before writing the code.

The app was deployed to users devices for them to test. Feedback was taken on the overall app and changes made as required. It was then compiled, uploaded and testing via the Android and Apple Stores. Finally the app was released to the public.

Plan for the future

While going through this whole process I was also thinking about an online app that would receive and present the data from the mobile app. This was the natural evolution because the internal database that the Gardens are using is accessible only on the local network. I had a pretty good idea of how this would all work before I even started building the mobile app. Flutter was nearly ready for producing web apps and I was hoping to reuse a lot of the code from the mobile app. Thinking about all this in advance can help you decide if the tech you are using is suitable not only for the current project but any extensions that may happen in the future.

Sorry it was such a long read! Hope it helped.

If you are interested in trying out the app it’s available on the Playstore and App Store as ‘Restore & Renew’ by Brendan Wilde.

Please follow and like us:
Facebook
Facebook
Instagram

Leave a Reply

Your email address will not be published.