Early Development and Design of Crowdsourcing Agriculture Mobile Application for Lambsquarters

As big data has become increasingly necessary in modern farming techniques, the dependence on high quality and quantity of ground truthing data has risen. Collecting ground truthing data is one of the most labor-intensive aspects of the research process. A crowd-sourcing platform application to aid lay people in completing ground truthing data can improve the quality and quantity of data for growers and agricultural researchers. In this study, a human-centered design process was followed including initial interviews with lo-cal researchers, focus groups with residents, and design evaluations with wireframes. The initial interviews with researchers at the University of D.C. revealed the need for ground truth data on lambsquarters in the D.C., Maryland, Virginia (DMV) area. Focus groups with users in the DMV demonstrated that the greatest motivation for the participants was having opportunities to develop their skills and access to educational resources. From the focus groups, design personas were created and wireframe prototypes were produced. The wireframes were evaluated by users using the System Usability Scale (SUS) and qualitative feedback. The wireframe design received an average score of 75.95, which indicates an acceptable design. From the feedback of the users, a mobile application will be created. This study demonstrates that by involving the end users, be they consumers or growers, in the design process of precision agriculture, their needs and goals will be considered and reﬂected in the precision agriculture tools they use every day. By designing big data technologies with the grower in mind, we can improve data interpretation and correct implementation in the ﬁeld.


Introduction
Modern farming has grown increasingly dependent on big data to improve the application of precision agriculture (PA) [1,2,3]. The type of data collected and used to create prescription maps for the farmers include soil types, weather, planting, and imagery [4]. While collected data about their farms is not a new development for the farmers, big data and PA has made it necessary to create new technologies to handle the millions of data points being used to design the PA algorithms for the prescription maps [5,6,7].
However, in order to train the PA algorithms, ground truth data, or geophysical parameter data, needs to be collected alongside the remote sensed data described above [8,9]. The collection of ground truth data is difficult as it requires a labor force which farmers and researchers simply cannot meet alone [10].
A human-centered approach was used in order to develop a crowd-sourcing application to aid laypeople to complete ground truthing for agricultural researchers in the District of Columbia studying lambsquarters growth. The purpose of this study was to conduct focus groups to gauge opinions on crowdsourcing initiatives in agriculture. A citizen science platform was prototyped for researchers at the University of the District of Columbia (UDC) who are interested in learning more about the growth traits of lambsquarters (Chenopodium album L.). Lambsquarters is an edible and highly nutritious weed found in North America [11,12]. While there have been sightings of the plant in the D.C., Maryland, Virginia area (DMV), not much is known about the optimal conditions of growth. By recruiting citizen scientists, DMV residents can assist researchers in reporting where and in what conditions the plant is growing, providing researchers with vast, rich data.

Precision Agriculture
Precision Agriculture (PA), also known as site-specific management, is the practice of managing field inputs for maximum yields. Industrial agricultural fields can range from 50 hectares (ha) to over 1000 ha in size, which means the characteristics of the soil can vary widely from one side to the other [1,2]. This causes the needs of different crops and soil to be different. The composition and health of the soil will determine whether farmers will be able to grow high-quality crops.
It is important for farmers to accurately identify what the needs of the fields are to maximize profits while minimizing waste. While traditional agriculture has the farmer apply the same amount of input (fertilizer, water, macro-nutrients, pesticide, fungicide, etc.) to the entire field, which can lead to areas in the field receiving too much or too little, precision agriculture uses techniques to identify what the specific needs are in each area for targeted application, thus reducing the overall amount of the input used, saving resources and reducing environmental impact [3]. Corn farmers who used precision agriculture technologies can save up to $25 per acre [13]. A survey of farmers in North Dakota found farmers who used PA technologies saved up to $1,500 a year per farm [14]. A study conducted by the USDA reveled that farmers who use any PA technology saves money compared to their counterparts who did not adopt the technology [15].
To implement PA, a cyclical process must be followed. The first step in precision agriculture is to acquire data such as soil characteristics, weather conditions, or disease and pest information. These measurements can be done with remote sensing using active or passive sensors or more invasive techniques by soil sensors or taking samples into the lab. Often this data must be coupled with locational data which will aid in the creation of a prescription map described in the third step. The second step is to process the data. Significant research has been done in this area of precision agriculture using big data analysis techniques. Much of this analysis is completed using statistical techniques or application software. The third step is to create a prescription from the analyzed data. This prescription dictates how much of an input should be applied in a specific location; this is usually in the form of a prescription map that is compatible with the applicator devices. In the final step, the map is integrated into the applicator devices, such as irrigation devices or fungicide applicators, which can follow the directions and vary the amount of input applied throughout the field. Once the application is complete, the precision agricultural process can begin again. The repetition of the cycle will depend on the input. For example, for irrigation, the cycle will be repeated on a weekly or daily basis [16].
A flow chart describing the precision agricultural process is in Figure 1. In this example, the steps of PA are illustrated as follows • Data Collection: spectrometer data is taken of Fuji apple tree leaves • Data Analysis: spectra analyzed for wavelengths that indicate the presence of Marssonina Blotch Disease • Prescription: analysis used to create a prescription map in ArcGIS • Application: prescription map used for the application of pesticide in the apple orchard One of the unique challenges of precision agriculture is that its data comes from a large area of land. While advances in remote sensing have made it possible to collect indirect data from large areas of land, it is still of vital importance to obtain the ground truth data. According to NASA, ground truth is "geophysical parameter data, measured or collected by other means than by the instrument itself, used as correlative or calibration data for that instrument data. It includes data taken on the ground or in the atmosphere" [9]. Without the actual identity of the vegetation, the disease, or pest in question, it will not be possible to accurately train the algorithm to correctly and accurately classify. Ground truth data is difficult to obtain because of the person hours needed to scourge the large area of land. Farmers and researchers just do not have the person hours to inspect all the vegetation they are surveying as needed for precision agriculture [5,17].
A solution to the lack of person hours is to crowdsource this aspect of the research with citizen science. Citizen science is a growing field in which non-professional scientists conduct their own experiments and participate in the scientific method and contribute to the scientific body of knowledge.

Crowdsourcing Initiatives in Agriculture
Citizen science has been used to bring in more laypeople, or non-growers, to be more mindful of environmental issues, or to help with classification of existing datasets [10,18,19]. One way that citizen science engages non-professionals in scientific research is through crowdsourcing [20]. In crowdsourcing, participants make small contributions individually while a central management system conducts the analysis on the collected data [21]. Examples of crowdsourcing projects in other domains include eBird, a collection of georeferenced bird observations from volunteer bird watchers and Foldit, an online game where players solve puzzles about protein structures to assist researchers in learning about protein folding [21]. In agriculture, crowdsourcing projects have become popular for plant breeding research [21]. According to Minet, et al, crowdsourcing is "(1) the realization of specific tasks or (2) the collection of data, information, or knowledge by a network of persons (the contributors) that are not doing so for their normal professional activities" [18]. Minet, et al. categorizes the current crowdsourcing applications in agriculture into 4 areas: 1. tasks such as solving math problems or classification, 2. local visual observations such as ground truthing, 3. data from disseminated sensor measurements such as leaf area index, and 4. knowledge such as Q&A forums and wikis [18].
The proliferation of mobile phones has also made it easier to recruit farmers to submit their observations for research [22]. A summary of crowdsourcing applications from [18] is illustrated in Table 1.
With the ease of reaching and recruiting participants, the question has arisen if the data collected is of a high enough quality to use for agricultural purposes. One study found farmers are able to make observations on their farm which match those of an agronomist at least 77% of the time, which indicates the farmergenerated data can be used accurately in crowdsourcing agricultural research projects [21]. In another, while it was found that individual farmers had low convergence, when the observations were aggregated, the convergence increased to be agreeable with expert data [23]. Recent citizen science projects have been able to harness the power of a large number of farmers by having them carry out experiments on their farm and crowdsourcing the results [21]. Examples include Crop Land Extent which had users in the Global South help determine agricultural field boundaries and Open Foris Collect Earth, a tool used by local experts in Tanzania to map forests [24,25]. While there are crowdsourcing initiatives for various aspects of precision agriculture, in the literature there are not currently any initiatives for the ground truthing task [10,18,4,5]. In this study, a human-centered design process will be used to design a ground truthing device.

Human-Centered Design
From an agricultural engineering standpoint, human-centered design is still an unknown and novel concept. As exemplified from the precision agricultural technologies surveyed in [26], agricultural engineers simply design and test their devices themselves and give the technologies the title of "usable"; if they can easily use it in the field. This approach misses the mark of what makes a device "usable" if the intended target user does not have an opportunity to use it or provide feedback of the design. Then they may struggle to understand how to use it in practice, since they lack the designers' intimate knowledge of the device. Ferrández-Pastor et al. discusses this in their study where they proposed a structure to include the expertise of the farmer in the design process [27]. Their call for the integration of UCD in precision agriculture is echoed by Lindblom, et al. who discussed the importance of including farmers in the design process for decision supporting information and communications technologies (ICT), also known as decision support systems (DSS) [28]. It has been shown that DSS systems support farmers as they strive for sustainable development. However, current DSS systems on the market fall short in fully supporting the farmer in their goals. Lindblom, et al. blame this failure on the design of the DSS systems, which incorporate data that scientists thought was important for the farmer, and did not include farmer's actual needs in the design process. This failure, along with poor user interface design and their perceived problem of complexity has contributed to the "gap of relevance" which can be bridged by human-centered design [28].
Zaks and Kucharik have also discussed the importance of including end-users in the development of agroecological monitoring infrastructure. In their paper reviewing the design of these infrastructures, they noted that the poor data interpretation of the user was not because of the sensor technology itself, but the lack of integration into the agricultural management tools end-users were familiar with, which would present the output data in a familiar format for the farmer. If the end-users could interpret the data more easily, then it would be easier to make decisions that mitigate the negative impacts of agriculture on the environment [29]. They acknowledged that to make the most robust systems possible, it will take collaboration with the end-users, policy makers and researchers in order to achieve our environmental and agricultural goals with these technologies. It is with these ideas in mind that a human-centered design process was used for this study.

Lambsquarters
Lambsquarters was the plant of interest in this study. Lambsquarters, or pigweed or goosefoot, is a fast-growing weed [30,31,32]. While previously lambsquarters has been spelled "lamb's quarters" or "lamb's-quarters", the current naming convention from Integrated Taxonomic Information System (ITIS) and United States Department of Agriculutre (USDA) is to call it "lambsquarters" [33,34]. Its scientific name, Chenopodium album L., can be broken up into it's origin: chen is Greek for goose, podos for foot, and album for white; this refers to the shape of the leaves and the color of the leaves' underside [32]. Older plants can take on a reddish color [35]. It grows to about 2 feet high but can be found as tall as 6 feet in rich soil [36,32]. It flowers in late spring into the autumn [36]. It is an unwanted weed that competes with crops, such as soybean, corn and tomato, and is often responsible for those crop losses [30,31,37].
Lambsquarters is a general summer plant [38,39]. It is found all over the world [38,39]. It needs sufficient light and low temperature to germinate [38,39]. However, it has been found that the germination rate of C. album is low, which means the wide spread of its seeds are needed to increase the chance of germination [39]. One plant can produce 100,000 seeds that can remain dormant in the soil for up to 40 years [35].
Lambsquarters is also known for its medicinal properties around the world. In Pakistan, it is known as a treatment for liver diseases and abdominal pain [45]. In India, it is used to improve appetite, as a laxative and as a diuretic [46].
Researchers at the University of the District of Columbia were interested in learning more about where in the district lambsquarters is grown. There have been sightings of lambsquarters growth in the D.C., Maryland, Virginia area, but not much is known about the optimal conditions that lead to its growth. Additionally, because of its nutritional value, knowing more about the best conditions for lambsquarters will help UDC researchers in advising local urban farmers how to best cultivate it. As a resilient plant, lambsquarters has the potential to be a great source of nutrition for residents of the DMV that they could grow on their own. This research on lambsquarters will help the University of the District of Columbia fits into their research agenda of climate change mitigation and food security. The University of the District of Columbia (UDC) is a public university in DC, and the only urban land grant university in the U.S. The College of Agriculture, Urban Sustainability and Environmental Sciences (CAUSES), which is UDCâs land grant arm, is focused on advancing the sustainability goals of the District of Columbia and the core objectives of the National Institute for Food and Agriculture of the U.S. Department of Agriculture. Through research-based academic and community outreach programs, CAUSES is committed to expanding knowledge of urban agriculture, urban sustainability and sustainable farming techniques that improve food and water security. CAUSES also has research and educational programs aimed at improving the health and wellness of local and global communities and mitigating climate change [47].

Human-Centered Design
The design process followed the human-centered computing methodology (Figure 3) is discussed in further detail in the sections below. The main ideas behind human-centered design are that the process is iterative and the user is in the forefront of the process from beginning to end. As seen in Figure 3 from [48], once it is established that there is a need for an intervention, the iterative process begins. This allows for further refining of the design if needed after the evaluation process. It also serves as a reminder to designers and engineers that the design process is not simply "finished" right after the evaluation, and the process must continue if necessary. The next sections explain how each of the steps in the process were completed for this study.

Identify the need for human-centered design
Based on previous research, we have identified a need for a human-centered design of a crowdsourcing application to aid in ground truthing for the design of classification machine learning algorithms for precision agriculture.

Understand and specify the context of use
Discussions with researchers at the University of the District of Columbia developed a need to map the growth patterns of the edible plant lambsquarters and to pair it with remote sensing data. As a land grant institution, it is part of UDC's mission to work with the community through several projects including extension classes, certifications, master gardener classes, and farmer's markets [49]. As it is also part of UDC's mission to educate the community in nutrition and gardening, working with citizens of the DMV as citizen science was beneficial.

Specify the user and organizational requirements
This step focused on addressing the opinions and preferences of users in the DMV area as a citizen science platform was created. In order to understand their needs as well in assisting with the data collection on lambsquarters, focus groups were organized. From the focus group results, design personas were created as the focal point of the initial platform design.

Produce design solutions
Wireframes of the platform were created and evaluated by potential users. From their feedback, mobile apps and a web-based platform will be created for users to upload ground truth data of lambsquarters.

Evaluate design against requirements
Users studies were conducted to evaluate the usability of the platform and evaluated using the System Usability Scale and qualititaive feedback from the users.

System satisfies specified user and organizational requirements
At the end of the evaluation stage, if it is not shown that the design meets the requirements, the process will repeat over again as described in Figure 3. If the requirements are met, then this step will be complete.

Focus Groups
Focus groups were conducted in order to specify the user and organizational requirements as part of the human-centered design process described above. Researchers and volunteer coordinators at the College of Agriculture, Sustainability & Environmental Sciences (CAUSES) at the University of the District of Columbia were consulted about the best locations and times to recruit participants and hold focus groups. It was decided to hold 6 focus groups in 3 different locations (UDC Van Ness campus, UDC Firebird Research Farm in Beltsville, MD, and the Arlington public library in Virginia). To provide easy access to participants in the DMV area, the focus groups were held after working hours or on weekends. Participants were encouraged to register before the focus group, but that was not required.
Focus groups began with participants reviewing the consent form and signing in using the UDC event sign-in sheet. A smart phone was used to audio record the focus groups. Sessions began with an overview of the terms crowdsourcing and precision agriculture and a description of lambsquarters, its cultural and nutritional importance, and the goals of the research project. Artifacts were used, in the form of a PowerPoint presentation, which displayed examples of crowdsourcing initiatives in agriculture, images of lambsquarters, and a description of the precision agricultural process. Following an interview guideline, participants were asked about their willingness to participate in crowdsourcing initiatives, barriers to participating, and features they would like included. Example questions include • By a show of hands, how many of you are familiar with precision agriculture? With crowdsourcing?
• Does a social aspect to the initiative matter in your willingness to participate?
• Does having a competitive aspect to the initiative affect your willingness to participate?
• What are some barriers that would prevent you from participating?
The sessions ended with an opportunity for participants to ask final questions or give final comments not covered by the previous questions.
The audio recordings were transcribed using otter.ai and coded using MAXQDA following a codebook based on the motivations of participation in crowdsourcing. From the results of the focus groups, design personas were created.

Wireframe Development
The prototype was created using Balsamiq and InVision. The goal of the prototype was to collect groundtruth data on lambsquarters. In order to achieve this goal, the prototype needs to (1) teach users how to identify lambsquarters and (2) have users submit images and qualitative information about lambsquarters they find. Researchers at UDC informed the details the app asks the user for about lambsquarters. The user-inputted details will help the researchers understand the different environments lambsquarters is found in the DMV. Based on the feedback provided by participants in previous focus groups [50], the prototype was made for a mobile phone. The flow of the mobile screens as depicted in the wireframes is available in Figure 4. The prototype was uploaded to a Samsung Galaxy S4 SGH-I337 for the user evaluations.
Based on previous focus groups that laid the groundwork for this research [50], it was determined the best method for recruiting the relevant users was through the weekly farmers market hosted by UDC. Researchers set up a stand at the farmers market for 3 weekends. Users who had previously participated in the research were notified of the next step of the project. The rest of the participants were recruited from the consumers and vendors at the farmers market.
Design evaluation sessions began with participants reviewing the consent form and filling out demographic information. Then they were instructed in the context-of-use and the task for the design evaluation. Participants were given the prototype to complete the task, with the researcher available for questions should any arise. Afterwards, participants filled out the System Usability Scale (SUS) with the opportunity to provide written feedback.

Focus Groups Results
Six focus groups were conducted over a period of 1 month. 2-3 participants comprised the focus groups, while one group was cancelled due to no shows. A total of 12 participants contributed to the focus groups.

Demographic Results
Of the 12 initial participants, 4 identified as African American, 6 identified as Caucasian, 1 identified as Hispanic and 1 identified as other. 7 participants identified as female and 5 participants identified as male. 7 participants reported obtaining a master's degree or higher. 2 participants had a bachelor's degree and 3 participants had an associate's degree. Participants also self-reported which neighborhood or state they resided in. Washington D.C. is divided into 8 wards for city planning purposes as illustrated in Figure 5 from [51]. 1 participant each were from Ward 1, 2, 3, 5, 6, 8 and 3 were from Ward 4. 1 participant was from Maryland, and 2 did not indicate where they were from. The University of D.C. is in Ward 3. The most frequent age range of the participants was 50-59 years old.

Coding Results
Categories for coding were created using the 4 primary motivators of participating in crowdsourcing from Brabham [52]: payment, job market signaling, competence development, and social affiliation. Overall frequencies of coded segments are in Figure 6.

Payment
Some successful crowdsourcing initiatives rely on paying their participants, such as Mechanical Turk. In these focus groups, the theme of payment did not appear very often, and when it did, it was in the context of monetary rewards related to their gardening hobby. An example quote is below: "What if you could get like private sector sponsors to offer rewards to people who reach certain levels, like, you know, $5 off at Home Depot, coupon or something?" Overall, it does not appear that payment is a motivation for urban gardeners in D.C.

Job marketing signaling
According to Brabham, some participants are motivated to contribute to a crowdsourcing project if there is a potential to leverage the experience into a paying job [52]. No participants discussed any motivation to use the project as a steppingstone to employment.

Competence development
The second-largest percentage of codes fell under this category. Most participants were motivated by the opportunity to develop their gardening and agricultural skills. Participants spoke about a willingness to use the crowdsourcing platform in order to learn more about lambsquarters and as an introduction into foraging.
Participants spoke of not only reaping the rewards of skill-development by using the crowdsourcing application, but they also suggested opportunities for further education such as exclusive foraging or cooking classes compensation for participating.

Social affiliation
The largest percentage of codes fell into this category. Many participants spoke of altruism, of being more willing to participate if it was communicated that the project would help the community in some way. An example quote is below: "If people have an understanding that this may help feed people that are, you know, in food deserts, for instance, you know, it's more than just fun" Participants also spoke about building social networks in person, of wanting to use the application and contribute data if it meant they could better connect with the research community or with their neighbors. These themes are explored in the next sections.
Connection to research community. Participants spoke of the lack of communication and the scarce information they are given when they volunteer for research projects. They spoke of researchers not telling them how their contributions would be used in the future, not following up to thank them for their participation, and not knowing what the bigger picture of the research was. Participants expressed a feeling of unappreciation.
Context was important to many participants. They stressed that if they were told the crowdsourcing application was for a college student and would help the community, they would be more inclined to help.
Connection to community. Participants spoke of the difficulty of finding people who enjoyed the same gardening and agricultural hobbies as them and saw potential in the crowdsourcing platform to help create those connections. Participants suggested using the application to advertise social gatherings or organizing group efforts to collect data at the same location/time. They also wanted to use the application to organize educational events where they could learn from their peers, or seed swaps so they can expand on their gardening skills. An example quote is below: "You see like minded people with the same passion, the same zeal, you compare and contrast your idea... You learn something you learn better when somebody with the same mentality explains something to you."

Data quality
A concern that was brought up in all the focus groups was how to ensure data quality. Because the crowdsourcing platform deals with an edible plant, there was concern over misclassifications that could lead to an individual becoming ill or harmed by interacting with a plant that is potentially dangerous.

Other suggestions
Overall, the focus groups were dynamic and produced many ideas for improving the participation of the crowdsourcing platform. Participants suggested having a day or two specifically dedicated to collecting the ground truth data about lambsquarters. By having a finite timeframe for data collection, instead of an open-ended call for data collection, participants believed it would improve the chance more people would contribute. Also, they were not opposed to having notifications and reminders sent to them.

Personas
From the discussions and themes that emerged from the focus groups, design personas were created. Design personas are "fictitious, specific, concrete representations of target users" [53]. They are created from the observable patterns in the data and and common behavioural characteristics of the target users [54]. These personas are then used to drive the rest of the design process [55]. The two personas are in Figure 7 and Figure 8. The design of the prototypes will be done with these ideal users in mind.  These design personas represent a conglomerate of the ideal user for the crowdsorucing agriculture application for users in the DMV. Primary goals of the users included learning to grow and cook healthy food, improving knowledge of local vegetation and gardening techniques, and connecting with likemindned people who enjoy improving the quality of food in the community. The primary frustrations shared in the focus groups covered technical issues such as the "single-purpose app" issue that could potentially slow down their devices and eat up storage. Both design personas were active in the extension community of UDC, whether by taking classes or being an enthusiastic volunteer for the university's agricultural projects. These traits and needs were kept in consideration of the design of the mobile application.

Wireframe Results
During June 2019, 6-8 participants evaluated the prototype at the farmers market on each of the 3 weekends. A total of 20 participants evaluated the design.

Demographic Results
Of the 20 participants, 11 identified as Caucasian, 5 identified as Hispanic, 2 identified as African American, 1 identified as Asian, and 1 identified as two or more race. 12 participants identified as male, 7 identified as female and 1 identified as other. 11 participants had a master's degree or higher, 5 had a bachelor's degree, 1 had one or more years of college, 1 had an associate degree, and 1 had some college credit less than a year. 8 of the participants were age 18-29, 2 were 30-39, 5 were age 40-49, 2 were 50-59, and 3 were 60-100. 5 participants were from Ward 3 and Maryland. 4 participants were from Ward 4. 3 participants were from Virginia. 1 participant each were from Ward 5, DC, and Florida.

System Usability Scale Results
As per industry standard, the System Usability Scale (SUS) was used to generate a well-accepted quantitative description of the usability of the mobile application [56]. It is a survey with 10 Likert-scale questions that has been calibrated to evaluate various types of systems for usability. The SUS gives a score from 0 to 100 where any score below the 70s indicates concerning usability issues, a score in the 70s is considered an acceptable design, a score in the 80s is considered a good design, and a score in the 90s is considered an exceptional design [57]. The SUS score was calculated using the methods described previously [58]. The average score of Mission LQ was 75.95 and the median score was 77.50. A histogram of the SUS scores is in Figure 9. A distribution of the scores by age group is in Figure 10. Based on the standard interpretation of the score, this design is acceptable, but there is room for improvement. Looking into the qualitative responses from the participants details what they found difficulty with and gives insight into where the design can be improved. More than 1 participant wanted to be more aware of the overall goal of the application and the project. They wanted the why of the application to be more obvious and communicated through the application.
Other participants mentioned how having the application on several screens was confusing. They did not know what was coming next or what would be expected of them. It was suggested to make the application 1 long form so the user can easily scroll through all the information at once.

Discussion
The findings suggest that community building is the most important motivational aspect for participants in the DMV. Whether that is community building with their neighbor and fellow residents in the district, or connecting with the research/academic community, participants expressed thriving on helping and getting to know each other. They are motivated by a feeling of being a part of something bigger and to be connected to others who share a similar goal or hobby. They also expressed a limitation on their time and interests; that they should not be taken for granted and their time and contributions should be respected. Thus, they suggested that the data collection be done on a specific and communicated timeframe, and that they should feel appreciated for their contributions.
Data quality was a concern for the participants and for the researchers as well. Processes will be included in the application development to ensure classifications of the vegetation is correct. First, volunteers will go through a training session where they need to make a few correct classifications before they can contribute their own original data. Images will be collected for each data point and an expert can monitor the classifications as they come in or spot check to address a specific data point at the request of the volunteer.
The result of the wireframe evaluation indicated areas of improvement in order to better meet user's expectations. Overall, the system at its current state was scored high by the participants. It is an industry standard that a SUS score greater than 80 is considered a good system, and the prototype averaged a score of 75.95. With minor modifications based on the feedback of the participants, the score can increase.
The age group that gave the lowest score, 30-39 years old, is also the group of participants who expressed wanting to know more about why the application and the research matters. This sentiment is an echo of the results of the previously conducted focus groups [50].
However, there was a minority of participants for whom the system did not work. In discussion with participants after the evaluation, some confessed that they did not own a smartphone and would not be able to contribute to the project, even though they found it interesting and wanted to. This also echoes a sentiment from the focus groups where participants who did not own smartphones offered to take written notes of lambsquarters when they found it and send the data electronically on their home desktops.

Conclusion and Future Work
The purpose of this study was to discover the motivational factors that would encourage residents of the DMV to participate in a crowdsourcing initiative to collect ground truth data of lambsquarters growth in the district and build a citizen science platform to crowdsource the data collection of lambsquarters growth in the DMV. Focus groups were held over a 1-month period on UDC's main campus in DC and the UDC Firebird Research Farm in Maryland. Participants shared their willingness to contribute to a crowdsourcing initiative that valued their time and efforts and where the researchers clearly communicated how and when their data would be used. Participants anticipated the platform to be used to create another social network for them to share and learn from each other's gardening skills. It was also suggested that processes should be included to ensure data quality was adequate.
From the results of the focus groups, design personas were created and a wireframe prototype was designed. The prototype was evaluated by consumers and vendors at the UDC farmers market. The participants gave the prototype an average score of a 75.95 on the SUS scale. A mobile application will be built based on this prototype and the feedback from the participants. Based on this finding, an alternative method of data collection will be created in addition to the mobile application. The alternative method will be based in Qualtrics.
From that feedback, a mobile application will be created and evaluated with user studies in the DMV. Lambsquarters data created from the user studies will be given to the researchers at UDC for further study. A pilot study will be conducted with the cultivated lambsquarters at the UDC Firebird Farm. Then, further user studies will be conducted at the public parks and other open lands throughout the DMV to collect the ground truth data of lambsquarters.