Organizing a multi-track virtual conference with Microsoft Teams Live Events: a technical playbook and lessons learned

June 29, 2020

How we organized the Galactic Collaboration Summit and lived to talk about it.

Due to the COVID-19 pandemic the European Collaboration Summit (a yearly, in person conference) has been moved to the autumn. We decided to organize a virtual, online conference, around the original conference dates. We chose 2nd and 9th of June 2020 – two days separated by a week (which turned to be extremely prudent!): the Galactic Collaboration Summit was born.

Prerequisites and pre-production time

Let us first briefly review the main organizational decision, which strongly influenced certain technical implementation choices and constraints.

Firstly, we knew that we had very limited resources, that it would be only four of us doing everything. Organizing, preparing, marketing, and in the end producing and delivering the whole conference. Even though the team has decades of experience with creating top-notch in-person conferences, an online conference was something new.

We were also aware of the many technical and human issues which could arise with such an event  (microphone or camera does not work, the screen resolution is too high or too low, speaker appears 5 minutes late, producer forgets to unmute the right speaker, awkward waiting in front of the empty screen before the session begins) and that fundamentally “too many cooks, spoil the broth”.  We also specifically wanted speakers from across the world, from many time zones. That meant, that for some of them, they would be delivering sessions at 3am. Not only an unreasonable request, but an extremely bad idea in terms of quality, and no fun for the speaker!

We really did not want the event to be an example of how NOT to do things and were fundamentally focused on delivering a great experience for our attendees.

Therefore, a decision was made relatively quickly, that we would go the other way, and have all the sessions pre-recorded and then professionally post-produced. The interactivity part would then be covered with (a majority of) speakers being present in the Q&A chat along with a live opening and closing.

This decision had several implications for us, the organizers:

  1. We had to receive the recorded sessions well ahead of time. We needed to chase all the speakers to deliver their sessions in time (at least one week – ideally more – before the event, so that we could post-produce them). Lessons learned here: some of them did not, and there were way too many long hours trying to get everything complete in time. There was a wide variety of quality as this was a new experience for many speakers, and numerous problems, in particular audio, needed to be corrected.
  2. Post-production should not be underestimated. For us, post-producing meant adding video intros with a jingle that includes speaker’s name, title, session name, and sponsor message, adjusting volume level, and appending the Collabsummit + Cloudsummit outro video to each session. Depending on the quality of the original recording, this would take 15-60 minutes per session. It meant filtering audio noise (when necessary and where possible), equalizing different audio levels with different speakers – especially if there was more than one speaker in the session, and their audio levels were not adjusted. With some sessions, also deleting the awkward pauses and “why doesn’t this work” comments. Of particular note, sessions recorded on a laptop with a noisy keyboard and/or trackpad are basically unfixable without many hours of work. This work was done in TechSmith Camtasia. The final rendering of post-produced sessions would take 20-30 minutes, depending on the session.
     
  3. Produce break commercials. There were 10- and 15-minute breaks between the sessions. From experience, if there is nothing happening in the track channel between two sessions, people will leave. The silence just feels awkward. Luckily for us, we had sponsor ads (oh, our sponsors were nothing but awesome as usual!), we had our own ads, and very important, we had created looping holding videos (10 seconds) for each track (room), where the program for that track and that day was displayed. These holding videos are extremely important for two reasons: they always indicate to the audience what is next in this room/track, and, to the producer, they can help to increase/decrease time in the playlist for the current track (more about that later).

From our experience, together with rendering, it takes approximately 60 minutes to produce a session. We had 60 sessions altogether, so that meant 2.5 days of producing time. Since the person who is doing that needs to sleep and to eat sometimes, and probably has some other tasks on her hands as well, it will take a week. In addition, the time within Adobe After Effects and Camtasia (or whatever is being used) to create the intro-jingles, in-betweens, ads, holding screens. That took us another week.

Once we had this sorted out, we knew that we could deliver a professional looking online conference.


Creating Azure infrastructure and Microsoft Teams environment

Since we wanted to have a stable, and best possible internet connection, and a dedicated environment for streaming the sessions, the decision was quickly made to use Azure Virtual Machines for “going live”. Since we have had five tracks, that meant that we needed to create 5 VMs in Azure. We have put them all in a virtual network, so that they can communicate with each other.

For us, production team members, to be more independent and able to jump between tracks, streams, and sessions as necessary, we also created five Azure AD / Office 365 service accounts with a Microsoft Teams license. Those five accounts would then create the Live Events and be their owners and (main) producers. We named those accounts the same as we named the tracks, for easier management and recognition.

Setting up Virtual Machines

We needed to have five identical VMs, where team members / producers could easily jump between them, perform the tasks as necessary, and work as efficiently as possible.

We created an Azure Resource Group for everything related to the Galactic Collaboration Summit, and all the resources were placed in that RG – VMs, Networks, Hard Disks, etc.

For the VMs, we used the “F16s_v2” template (CPU optimized), with 32 GB RAM, high performance SSDs, and the latest Windows 10 version installed. We used the same admin username and password for all of them, fully knowing that is not a best practice. As there was no sensitive data on these machines, and the whole Resource Group will be physically destroyed after the conference. The lesson learned here is that at the time of writing, it is still not possible to create those VMs in the German data center, so we chose West Europe (Netherlands/Ireland).

We used Video Lan VLC media player for feeding the Teams Live Event stream.

The following steps were performed for each of the VMs. If a different media player is used, these steps need to be adapted accordingly.

  1. Create the virtual machine using the template (F16S_v2 in our case) and the latest Windows 10 operating system.
    • Name the machine after the conference track.
    • Set the administrative username and password.
  2. Install the new Microsoft Edge browser
  3. Add the service account, which has been named the same as the track (and as this machine) as Work or School Account to that Windows 10 machine. Use the Settings => Accounts => “Access Work or School” option in Windows 10.
  4. Go to Settings=>Notification Settings and remove all notifications on the machine. All of them! Off! 🙂
  5. Change the time zone of the VM to the “main” time zone of the conference
  6. Check, and if necessary, adjust the power options, so that computer will never shut down or turn off the screen.
  7. Disable all system sounds, by going to Control Panel -> Sound -> Tab “Sounds” -> Sound Scheme: No Sounds.
  8. Pause all Windows Updates until after the conference (14 days were enough in our case) by going to Settings => Windows Update => Pause Windows updates
  9. Install the playlist creation software, that enables production rundowns creation. More about this in a later section.
  10. Install the VB-Audio VB Cable software and create virtual audio devices. More about this in the next section
  11. Set unique desktop background for that machine, with the name of the track, so that it is immediately recognizable which machine is it.
  12. Download and install the Microsoft Teams. Login into that client with the service account dedicated to that machine.
  13. Download and install the Video Lan VLC media player
  14. Turn off all on-display messages in VLC, by going to Tools => Preferences => “Subtitles / OSD” tab => uncheck “Enable On Screen Display”, “Show media title on video start” and “Enable Subtitles”. This will prevent VLC of displaying too much unnecessary information each time the new file is loaded.
  15. Disable the full screen controller in VLC (which displays controls on the bottom of each video on mouse over, or on video change), by going to Tools => Preferences => Show settings “All” => Interface => Main Interfaces => Qt => uncheck “Show a controller in fullscreen mode”
  16. Disable playlist randomization and all other Playlist “quirks”, by going to Tools => Preferences => Show settings “All” => Playlist => uncheck EVERYTHING except of “Automatically prepare items”.
  17. Download all the sessions, in-betweens, and holding screen videos into a folder structure, which will be identical on all Virtual Machines (“C:\GCS\…” in our case). We used a SharePoint Online library as a central storage for all our materials, synchronized it with OneDrive for Business, and that worked very well.

Solving system audio issues

As is well known virtual machines have no audio devices available or configured. They can usually mount and use audio devices from the host machine. In our case, we needed something different: we needed these machines to play sound via a “loopback”, so we needed to install a virtual audio device to achieve that.

Fortunately, two guys called Vincent Burel and Jean-Sylvian Loezic have created a nice “virtual audio device”, called VB-Cable (www.vb-cable.com), which simply makes a loop back – all audio coming in the CABLE input is simply forwarded to the CABLE output. So, even if there are no physical audio devices, such as microphones and speakers, this virtual machine will be able to play the sound to that cable “device”.

The free version of VB-Cable is enough for this purpose, but since it is donationware, it would be nice if the work of those two guys could be supported by “buying” one of the advanced versions, and donating anything between 5€ and 25€

Important notice here:

In order for this to work, the RDP connection properties must be set so that audio playback is played on the remote computer, and not on the local machine. This is done by editing the RDP file (which can be downloaded from Azure). In the “Local Resources” tab, there is a “Settings” button in the “Remote audio” section. “Play on remote computer” option needs to be selected. The RDP file must be saved before closing and connecting to this machine.

This setting must be saved permanently in the RDP connection, and used for every log in. Setting it only while installing the VB-Cable software is not enough.

Once this is done, the “VBCABLE_Driver_PackXX.zip” for Windows can be downloaded from the web site and installed on the virtual machine. A restart will be required.

Once installed, the “loudspeaker” icon in the system try should be enabled, and in the Control Panel => Sound => “Playback” tab, “CABLE Input” should be listed as the default audio device.

 Setting up Teams Live Events

Before we start with the creation of the Live Event, we need to make sure that the Live Events can be broadcasted to the general public, and not only to people in our organization (which is the default setting).

In order to do this, the administrator with appropriate rights will have to change the existing Global Live Events Policy, or to create and apply a new one. That policy can be found and changed in the Microsoft 365 admin center => Teams admin center => Meetings => Live events policies. From there, either the existing “Global” policy can be edited, or a new policy can be created and applied.

In each case, the “Who can join scheduled live events” should be set to “Everyone”.

Please keep in mind that the policy change can take a few hours to take effect.

Once this policy has been set, live events for general audience can be created in the Teams desktop client. The option to do that can be found in the Calendar section.

 

 

Please note that at the time of writing, Teams Live Events are not yet enabled in German data center, so you will need Teams hosted in other regions.

As mentioned before, for the Galactic Collaboration Summit, we are using service accounts to create the Teams Live Events, and those service accounts were also the main organizers. We have then added to each event our team members, either as a producers or as presenters. That way, our team members could easily jump from event to event to make sure that everything is fine, without need to worry if other producers are in the event – we could be sure that our service account producer will always be present in that event.

Here is a comparison table between producer and presenter permissions, in order to decide which of the team members should be set as a producer, and who will be “only” presenter.

Capability

Producer

Presenter

Select video feeds of other presenters and send to event

Yes

No

Start live event

Yes

No

End live event

Yes

No

View event preview (stream that’s sent to attendees)

Yes

No

Manage recording and reports

Yes

Yes

Join as attendee

Yes

Yes

View live attendee count

Yes

No

Chat with other producers and participants

Yes

Yes

Share screen into the live event

Yes

Yes

Share system audio

Yes 

Yes

Invite users to join as presenters

Yes

Yes

Mute all other presenters

Yes

Yes

Moderate Q&A

Yes

Yes

Once we selected our producers and presenters, we needed to decide about a few other Live Event options, such as visibility and reporting options.

As shown in the screenshot above, we decided to go public with our live event. Since we will have one long Live Event per track / day, we decided against making the recordings available to the attendees (we made the individual session recordings available later using a different mechanism), and we decided against the captions, since they mainly don’t work that well in technical sessions, especially as the overlays can be distracting during demos, and for those who are not native English speakers.

On the other hand, we definitely wanted to have the Attendee engagement report, and Q&A.

We strongly recommend creating a test event before the real event, with the same settings as the real event, where you can test all the scenarios described in this article.

The last screen of the Live Event creation wizard will create the event and give us the ability to copy the event link which should then be sent to the attendees.

Masking the attendee links

We strongly recommend that Teams attendee links are not sent directly to the attendees. They should be rather first masked with an URL shortener that supports editable URLs.

There are many reasons for doing this, here are some:

  • Original Teams Live Events attendee join links are very long. They are difficult to copy, to send, and they are very prone to error. If you paste them into a Word document or email, the autocorrect will even try to capitalize them.
  • With the short URL stubs, it is possible to offer your attendees a downloadable agenda well beforehand, way before actual Teams Live Events have been created. Those links can in the beginning lead to the event home page, or to some similar destination, and then later be updated with the real Teams URLs.
  • If something happens with the original URL (if Live Event has been prematurely ended/deleted/cancelled, or if some other error has happened), the organizer can simply edit the short URL, and update it with the new Teams Live Event URL, without the need to communicate that new URL to all of the attendees. The old, “short” URL will continue to work for them.
  • Post conference, those short URLs can be updated again, to lead to the session recordings (if those are offered).

We have used our own URL shortener at csmmt.eu, but a number of public URL shorteners with those capabilities can be found.

Note: we also used the short URLs within our event management platform, GatherUp, which provided the stream URLs to registered attendees.

Creating Production Rundowns

Creating a playlist and pressing the “Play” button in VLC (or other media player) seems like a trivial activity, and we should pay not much attention to it, right?

Unfortunately, it is not that simple. While creating a playlist really is a simple task, playlist calculation is more involved. For Galactic Collaboration Summit, we wanted the main media elements – prerecorded sessions – to begin at the times that were announced in the schedule (+/- 10 seconds). Since the sessions are of different length (we had everything between 20 and 35 minutes), and since we don’t always want to play the same ads in the same sequence between the sessions, the problem of creating the correct playlist becomes obvious. More so because every single change would lead to a complete recalculation of the playlist, and if done manually, that would take hours. Last but not least, such playlists are prone to errors, because if a small delay, or a slight problem occurs during the live event, the producer is faced with a serious problem, since recalculations on the fly are impossible.

This problem is fundamental to media and TV production. A production rundown (sometimes also known as a cue-sheet) is an item-by-item sequence of events that happen within a show, a timeline. An online conference is effectively a show, or in our case, five concurrent shows, and so production rundowns are an ideal mechanism to help manage the complexity and avoid anything being missed.

We started looking for software that could create M3U playlists, which can be used by VLC media player (or other media player), and would support the following:

  • Set the start date/time of the playlist. All the other times of individual media items would be calculated as an offset to that initial time. Changing this start date/time would automatically recalculate times of all individual items in the list.
  • Media files could be added to the list using drag and drop. Their length would be determined, and their position immediately calculated or recalculated.
  • Each item should have easily and clearly visible start and end time.
  • Rearrange items in the list using drag-and-drop.
  • Select one or many media items and duplicate them or copy them to the bottom of the list.
  • Load and save the production rundowns as M3U files.

The number of playlist makers which support these features, was exactly zero. Since we did not want to manually create those lists, the only viable solution was develop that software.

Admittedly it was a very quick and dirty development, completed in just over two hours, however fully functional, and thus saving countless hours of manually creating, calculating and recalculating the playlists.

As a bonus, when creating a M3U playlist, the tool replaces the file title with a custom string which combines the position, start time, and the file name. That way, even during the event, we would know exactly what is playing, if it is being played on time, so that we could react, skip a file, or do whatever we wanted, without the fear of “getting lost” and without need to recalculate the rundown.

Note: in order for this to work, media files must not include metadata, such as title, album etc. Otherwise, VLC will always show that embedded metadata, instead of showing the custom title that we have defined in the M3U list. If there is metadata present in your media files, you can easily use the “File Properties” dialog from the Windows Explorer to delete it. Furthermore, sometimes VLC will remember the “old” metadata in its own cache, so if that happens and you get weird titles in VLC, make sure that you empty the cache.

Our production rundown software was, besides Microsoft Teams, probably the single most important piece of software that we used during the event. It enabled us to notice and to correct the glitch on day one, when VLC had randomized the playlists, before any of the attendees could notice it. This enabled us to always “be on time”, and to have a full control of what is playing now, and what will be playing next.

In an ideal world, I would take 2-3 days to refactor and to polish this code, and to publish it on GitHub. This is not an ideal world. If you are organizing an event, and you need something like this, please contact us, and we will provide you the tool free of charge on a per-need basis.

Showtime!

Finally, the day of reckoning, team nervousness had reached peak! Playlists on all five virtual machines had been set to start at 10:30 am CET, and “play throughout the day”.

Starting the Live Event

In order to start the Live Event from the virtual machine, the producer (in this case the service account user) had to join the event in the desktop Teams client and click on the “Share” button. Even if there are no microphone and camera present on the virtual machine, it should be still doublechecked that the producer is muted and not sharing thier own video stream.

The next step is to share the VLC media player, and whatever is playing within it. The crucial thing here is to click the “Include system audio” checkbox on the left side: that way, the audio from VLC will also be broadcasted. This is why we needed VB-AudioCable as covered in the previous section.

The third and last step, is for the producer to send the shared VLC player live, for attendees to see it. In the screenshots below, VLC player is not in full-screen mode, and we recommend starting it like this (at least an hour prior to the first attendees joining), to easily control what is playing and to ensure the event is on time. When attendees start joining, VLC media player should be switched into the full screen mode by double-clicking the player screen.  

Whatever you do, do not press the “End” button here. It does not end the current playlist, or current window streaming, it ends the whole event, and it is not possible to restart it. It didn’t happen to us during production, but it happened twice during the warmup day.

The position of the “End” button, and that it definitively ends the Live Event, without the ability to restart, is a serious UI and functional issue with Microsoft Teams, and I hope that the team in Redmond will address it as soon as possible.

During the production

First things first: you cannot have enough screens! Our team members were working with multiple screens on the main computer, and with additional tablets to control and preview the Live Event.

This example shows three screens on the main computer:

Left Screen

The left screen has all five VMs which were used for production. A must have application here is “Remote Desktop Manager”, which enables easy control and switching between multiple machines. It also has quite a nice “Tiles” view, which enables monitoring all 5 VMs simultaneously, and to “click into” one of them if/when there was a need. That view is shown in the screenshot below.

Middle screen

The middle screen has the Teams Chat with the rest of the team. Why chat, and why not a team, or Live Meeting chatroom? Two reasons. Standard chats can be popped-out in a separate window, which is ideal during an event. And, we were all monitoring different Live Events, so we needed a chat that would be independent of those events.

Right screen

The right screen has the main desktop computer, which is logged in to Microsoft Teams with personal credentials, as a producer, and for monitoring one Live Event (one track).

This is used for moderating the Q&A in that track, monitoring that everything is going as planned, etc. Please note that production work is not done on the main desktop machine, it is done on the virtual machine under the system account credentials. That way the event team can “jump” between all five live events (tracks) without ever jeopardizing the main production process.

Trust is good, controlling is better

Besides the production environments, it is important to always be monitoring the event as it is being broadcast to attendees. This is where tablets are extremely useful, although these can be any clients. By using non-producer accounts, the end results can be monitored, and any buffering issues or similar glitches are readily apparent if they are local device or possibly a wider problem.

The following diagram shows our setup per track / per producer:

  • The main producer was controlling playback and adjusting glitches, logged as a service account on the VM.
  • Each track had one team member logged in using their own credentials as a producer or presenter, to control the track, and to moderate the Q&A, but not to do any production work.
  • The same team member was logged into that same track as a regular attendee on a mobile device, to monitor the attendee experience.

Adjusting on the fly

Nothing worthwhile ever happens without problems. We had to restart the playlists twice over the two days. These moments proved the wisdom of creating the production rundown software with exact on-air times – as it allowed these glitches to be easily dealt with, and with barely anyone noticing it.

Even if nothing else goes wrong, VLC will happen! There will be small delays when VLC switches between media files. This is invisible to the attendees and they will not notice anything untoward. However, the delays add up and quite quickly the stream is lagging 10-20 seconds behind the original schedule. This is another reason it is practical to have the 10-sec holding screen videos, because they can be easily skipped, and nobody will miss them. Thus, the time can be regained on the fly.

The issue here is how to do it, in a way that the attendees do not notice. As we had removed all the visual controls from VLC player, the only way would be to exit full screen, show the VLC chrome, and then skip to the next item in the playlist, and then re-enable full screen. That would work, but it would be immediately noticed by the attendees.

A more elegant way is to navigate media through the thumbnail in the task bar. When the mouse is positioned over VLC in the task bar, a thumbnail will pop up, with basic controls: pause/play, forward and backward. In the tooltip above the thumbnail, the currently playing file is displayed, with its expected start time (thanks to our customized titles!). It is then easy to wait until the next 10 second holding screen begins to play, and to then just “skip” it. This is how we could always easily gain back 10 seconds, without attendees ever noticing it.

Because we only shared the VLC window in Teams, it was easy to switch between the other windows and applications in that VM, such as rundown producer or Teams shell, to control the stream and make adjustments – the attendees will never see any of that work.

In the end: a love letter to Microsoft

From the beginning, we decided that we wanted to give a chance to Teams, regardless of how large the event was. With a few thousand attendees being simultaneously logged in at the peak times, we have to say that Teams has done its job very well, and that we had exactly one infrastructure problem (one event ended prematurely). All in all, it worked, and it worked very well!

However, there are few things that we have fed back to the product team, and some suggested improvement:

  • Ability to continue live event if it has ended prematurely for whatever reason. This is crucial. We had one event ending prematurely on day two, and we still do not know the reason (the most probable explanation is a technical glitch, with still a small probability of human error). In such cases, you cannot just “continue” the event, you need to completely recreate a new one, and then to send the new join link to everyone… (luckily, that last part was mitigated through having URL stubs from our URL shortener, so I could almost “hot-swap” the join links and reduce the glitch). The ability to resume a live event would make the life of producers so much easier.
  • Positioning of the “End” (live event) button in the UI. It is too large, too prominent, and too inviting to be involuntarily clicked, when producer just wants to leave the event, but not really to end it. By making this button smaller, or even better, hide it somewhere in the meeting settings. We clicked on it two times by mistake during the warmup day. The button should also include a “confirm” option (i.e. Are you sure you wish to end the live event).
  • Enable sending multiple screens/contents live. As only one screen or content can be “sent live” there is no way of producing side-by-side, or screen-in-screen content. Just think of all the panel scenarios, where multiple panelists should be on the screen at the same time. This actually made us decide against panels, despite planning them initially. Whilst a producer can switch between this feels clumsy, and it is always a few seconds late. This is a feature which is much more akin to a “meeting scenario” than a “live event scenario”.
  • Enable playing media files, and creation of schedule and production rundowns within Teams Live Events. Earlier in the article we described how we handled this with custom software and VLC. But how much better it would be if Teams could send media files from local (or cloud?!) storage directly into the Live Event? Eliminating the need for third-party media players.

    Similarly, if Teams could schedule a queue of files, with the exact times of when they should be aired, or the import of an existing M3U playlist. This would eliminate a need for any third-party production rundown software.

It is worth noting how responsive and keen the Microsoft Teams product group have been to both support the Galactic Collaboration Summit, and to spend time with us discussing ideas for future improvements. Their attitude and engagement with the community is a breath of fresh air in the industry.

Overall using Microsoft Teams was a great experience. If you are planning a live event, there really is not much that speaks against using it. It was stable, attendees had a good experience, there was very little “buffering”. It just worked.

This was our first Teams Live Event and other than a few minor, barely noticeable glitches, everything went extremely well. These glitches were mostly human error or issues with other software (VLC’s crazy default of randomizing playlists).

Chapéu, and thank you, Microsoft Teams!

Teams rocks, #communityrocks

Collaboration means two-way communication!

Did you like this article? Are you interested in this topic? Would you like to share your knowledge and experience with the authors and other readers? Our #communityrocks group on Facebook is bringing everyone involved together: authors, editors and readers. Join us in sharing knowledge, asking questions and making suggestions. We would like to hear from you! 

Join us on Facebook


Related Articles

Submit a Comment

Your email address will not be published. Required fields are marked *