May 01 2015

A Member-Driven Network – Part 2: Real-time Feedback

Hive HQ, The Hive Network Effect

Hive Chicago is a Member-driven Network.

As part of Mozilla’s Hive Learning Networks global initiative, Hive Chicago shares its mission, vision and emergent theory of change with all Hive Networks: mobilize educators in our communities to create connected learning experiences (that teach web literacy) and catalyze others to extend that work.

Yet, as a network of local organizations in Chicago, the unique goals, strategies and calls-to-action that the Hive Chicago adopts to advance that mission are directly informed by the collective aspirations, needs and challenges articulated by our individual, organizational, and community members.

In Chicago, members of the Hive collaboratively shaped a set of four goals in 2013: equitable access, learning pathways, innovative program design, and external value. To achieve these aspirational goals, Chicago membership further articulated six Moonshots in 2014. These are calls-to-action or issue areas that organize Moonshot working groups to generate solutions – onramps to learning, transportation, school engagement, parent engagement, youth voice, and data informed decision making.

The experience of exploring, creating and sharing collaboratively in the Network – either through meetups, online forums, blogs, funding opportunities, youth learning events, or other gatherings and venues for engagement – is also shaped to meet the needs of the network. Hive Chicago collects data on member activity, solicits experience feedback, and engages research partners to assess the effectiveness and to inform the design of our engagement platform and professional learning community.

What does that process look like? In this post – Part 2 of a 2-part series – we continue the conversation started in Part 1: NYU Research Recommendations, where we began by reviewing the key findings of a recent NYU research report on Hive. Here we look at real-time participation data collected over the last 18 months and consider how Hive Leadership uses all of this information to craft their strategic community engagements.


Meetup Feedback

In addition to the NYU Report, Hive Chicago continuously collects network feedback directly through surveys at network engagements and indirectly through tracking participation activity, event attendance, and organizational profiles. As a complement to the NYU key findings, Hive staff also shared samples of this data at the March 2015 Meetup as well. These are presented below.

Meetup Feedback collected to date includes over two dozen questions tracked intermittently over the 14 months (in grey). The Likert scale used ranges from 1 – Strongly Disagree to 4 – Strongly Agree. Bolded lines highlight some example questions:
#5 – “I think that Hive Chicago is on the right track.”
#11 –  “Today’s meetup provided me with an adequate opportunity to connect one-on-one with Hive Leadership”
#19 –  “Today’s meetup provided me with an opportunity to connect with someone new in the Hive, or reconnect with someone I otherwise see infrequently.”

We can draw a few conclusions from the data above. First, overall, our meetup attendees have found the meetup experience to be improving over the last year. This is implied by the up-and-to-the-right trend in the lines above, but clearer and more compelling in the full dataset. Second, the meetups currently and consistently do a better job at connecting our attendees to each other than they do at connecting attendees to Hive staff, though both have improved over time.

Finally, the sense that “Hive is on the right track” is significantly variable from meetup to meetup. This information is regularly used by Hive staff to gauge how attendees have received new direction or messaging offered by network leadership at meetups. It helps indicate when leadership plans are connecting to member expectations.

Meetup Attendance

In the anonymized meetup attendance spreadsheet below – click here to view the spreadsheet in it’s own window – each row tracks a single attendee and each column represents a meetup; cells are marked orange when an individual was in attendance, while those marked blue indicate they were not.

The meetup attendance data is very illuminating. The first thing to note is the “long tail” of attendees with intermittent or singular attendance. The sheet is sorted with highest attendance at the top, to lowest attendance at the bottom. You can see the long tail by scrolling downwards and noting the transition from predominantly orange to blue. Note that many of these folks were attendees to our hack day in January 2015, Hive Chicago Buzz

Secondly, when individual attendance is compared against member organization attendance the following conclusions emerge (note that member organization attendance is not shown here, but is calculated by considering when any staff person from an organization is in attendance at a given meetup):

There is a consistent representation of member organizations:

    • Our 64 Member Organizations attend an average of 50% of meetups (6) per year on average;

There is a variable representation of individuals:

    • 374 unique individuals have attended Hive Chicago meetups in the last 18 months;
    • 50% of those attendees came once but have not yet returned (modulo our next meetup);
    • 30 of those attendees attend 50% of meetups or more (6 or more) per year on average.

What is amazing about this information is that despite the fact that very few people attend every single meetup, there is a pervasive sense of connectedness in our community – reflected in the NYU report – that is driven by many other interaction opportunities outside of meetups: online in our member-forum, through project collaborations, and at other Hive events and programs. The key takeaway is that:

A thriving network can operate through distributed and loosely connected relationships while remaining tightly in synch.

This data also helps us to think strategically about how we design the meetup experience in 2015: if 50% of our attendees only come once, then we need to be much better prepared to make that single interaction a meaningful one. This is true whether or not there is potential for them to return! Ensuring that people who make the time to interact with us face to face just once have a clear sense of who we are is critical if we only have that one shot. First impressions are lasting.

This also provides us with an interesting opportunity to sample folks in different attendance groups to find out why they do or do not return and where else they may be connecting with the Hive in a way that suits them better. These surveys will be conducted in the next couple of months and their results will be invaluable.


Advancing the Network in 2015

Prior to the March meetup, Hive staff prepared a draft engagement plan and calendar for 2015, based on perceived network needs that were apparent to Hive staff from their experience in 2014. Before these plans were finalized, it was important to assess them against actual member expectations. This was another opportunity for feedback. After reviewing the NYU Key Findings and the meetup feedback and attendance data, Hive staff presented meetup attendees with their plan and the calendar shown in the spreadsheet below.

After reviewing the plan and calendar, meetup attendees were prompted to provide one final round of feedback:

After considering the NYU Key Findings from their 2013-14 Academic Year report, seeing the feedback and meetup attendance data collected by Hive Staff during that same year and hearing about the 2015 calendar for convenings that includes Meetups, Community Calls, and “Hive Dives”, what are your reflections on what should stay, what should go, and what should change for the Hive Experience in 2015?

Attendees completed a survey to identify one item of the Hive engagement strategy they would keep, one they would change, one they would add, and one they would throw away. Hive staff received over 30 individual survey responses and while no specific items dominated the keep, change, add and trash categories, the following general trends emerged after review:

  • Variety – people appreciate a variation in their experience, e.g. food, topics, formats, after hours, target audiences, even furniture format;
  • Skill Building – people want to continue having opportunities to build their skills, e.g. communication-tool-tutorials, skill/fail shares, RFP support, etc.;
  • Inclusivity – there’s a widespread desire to continue broadening inclusion of various forms of diversity, e.g. geographic, racial, formal/informal, etc.;
  • Mechanisms for Connecting – people need more information about digital tools already in place for connecting the community and training to use them;
  • Connecting to Resources – people are looking for a more user-friendly format that allows individuals to find and digest our network resources and learning products;
  • Connecting to Peers – peer-to-peer connections and exchanges are Hive’s biggest asset and value-add for the community;
  • Less Yack & More Hack – people feel like they understand what Hive is all about and now they want to spend more time working on Moonshots;

These are the Hive Chicago design criteria for 2015 community engagement.

Collecting data, from qualitative feedback to quantitative attendance, provides Hive Chicago leadership essential information that is used to drive decision making and to define the design criteria for our experiences. If our goal is to increase the quality and impact of our member engagements in service of our goals, then responding to the design criteria above are the key to our success. However, the process of collecting this information also reveals new areas of improvement where additional data could help us refine our practice. This is the power of our member feedback.


The post A Member-Driven Network: Research, Data & Feedback for 2015 appeared first on omnignorant.

Leave a Comment