Overview

The following diagram shows a typical flow of support issues from end users through to the technology operations team. 

End users and client groups

End users of technology services may receive support directly (e.g. via the technology service desk) or indirectly (e.g. general public contacting a contact centre). 

Those contacting the support team directly will do so via call, email, chat, service desk portal or messaging applications. They may also deal directly with onsite support technicians who will raise the issues on their behalf. 

The indirect end user groups will typically contact a one-stop-shop for all their needs. For example a potential visitor may call a contact centre regarding ticketing, transportation, parking, opening hours or any number of things. Or press agencies might contact a media helpdesk as their first point of contact. In a small number of cases the enquiry may relate to a technology service problem that the respective contact centre will raise with the technology service desk. 

 Direct end user groups:

  • Workforce from the organising committee including staff from all functional areas, including contractors.

  • Service provider staff engaged in operations, including partners and contractors delivering services for the event.

Indirect end user groups

  • Broadcasters – including the host broadcaster, rights holding broadcasters, media rights licensees. 

  • Media – accredited media including written press, agencies and websites, photographers and photo technicians, social media influencers and bloggers. 

  • Volunteers of the organising committee assigned to roles for the event. Many of these will also access to the technology service desk through their assigned event role, but typically have a primary point of contact for all their queries. 

  • Visitors, spectators, and general public – ticketed and non-ticketed. 

  • Others - depending on the event other groups may include athletes, teams, international federations, participating countries or committees, commercial hospitality

Something to think about. There may be several help desks or contact centres needed for the different user groups. While it can make sense to provide a one-stop-shop for the different client groups, ideally the same technology platforms would be used (ITSM tool, call/chat capability, contact management).

 

Technology Service Desk (L1)

 

The technology service desk provides a single point of contact for all supported services detailed in the technology service catalogue and for the agreed client groups. Contact channels include email, phone, live chat, self-service portal, walk-in, virtual assistant or messaging (e.g. webex, teams). They will log the incident or service request and try to solve it directly with the end user if possible. 

If the service desk cannot resolve the incident or request directly they will assign it to the next most appropriate technology support group for further investigation, diagnosis as defined in the technology service catalogue. This includes:

  • A venue technology team (including office locations) who will check the issue first hand and try to resolve it by assigning it to the appropriate technical team or person. E.g. an end user laptop will not start or local network equipment issue. 

  • Another dedicated technology support group responsible for the support of the particular service, where appropriate E.g. ticketing transactions are giving an error via the mobile app.  

  • In some cases the ticket may be assigned to the Technology Operations Centre if that is the most appropriate team to take ownership of the issue. 

 

Venue Technology (L2)

The venues (competition and non-competition) are where all the action takes place, so they will have an assigned technology team to address end user requests and incidents, resolve service failures and manage changes. It is expected that a large portion of end user tickets will be solved at the venues even if they are raised via the technology service desk. 

A typical venue technology team, for the event operations phase, may be structured as shown below. This obviously depends on the event type, venue size, deployed services and organisations involved. 

 

Roaming venue technology team

A team with domain technical experts that may be required to respond at venue level to incidents or requests. The roaming team may support the following activities:  

  • To provide technical capability and capacity to the venues without that capability. 

  • To provide additional technical capacity to the assigned venue technology team (reinforcement) during periods of high activity (e.g. competition/match day, deployment) 

Technology Support Groups (L2/L3)

There are many different teams involved in the support of event technology services. These may be application support teams, office IT system support or others as defined in the technology services catalogue. They may be located in supplier premises, organising committee headquarters or other locations. Some of these teams may sit inside the Technology Operations Centre. Regardless of the location of the technology support group, they all form part of an end-to-end support model using common processes and tools as much as possible. 

Technology support groups not located in the TOC or venues, consist of various application support, telecommunications & network support, remote operating centres and others that have responsibility for the operation, monitoring and issue resolution for services within their scope. They may be assigned tickets directly from the Technology Service Desk, venue or TOC and use the same processes, tools and SLAs as the TOC.  Offsite CSOCs, NOCs may fall into this category. 

Technology Operations Centre (L3)

 The Technology Operations Centre (TOC) is the facility where all key technology areas collaborate during the operational periods of the major event (including Test Events and Technology Rehearsals). 

The TOC is an escalation, support and coordination point for technology-related issues that cannot be solved by venue technology teams and for event-wide technology infrastructure and services 

Specifically, the TOC provides:  

  • A central location for end-to-end service monitoring across all venues and facilities 

  • A point of escalation for technical and operational issues that cannot be resolved on the venue 

  • A location to facilitate rapid resolution of critical incidents by having key decision makers and technical expertise in the same location 

  • Management of technology service management processes, tools and performance measurement

  • A single source for Technology communications to and from stakeholders 

 A brief description of the TOC support teams is provided below:

  • TOC Director: provides leadership and accountability for all technology services during the critical operations periods.

  • Critical Situation Manager: directs and coordinates the resolution of major incidents and other situations requiring cross-team coordination across all support groups ensuring timely resolution and effective communications. 

  • Staffing & entitlements: coordinates the technology workforce and logistics activities including shift staffing, accreditation, VAPPs, meals and transportation.

  • Venue Technology: responsible for the operation of technology services across all venues during critical operations periods and coordinates final venue technology readiness, transition, and decommissioning. All Venue Technology Managers (VTMs) report into this role.

  • Network & Telecom: responsible for the monitoring and management of all network & telecommunications infrastructure and services at venues, facilities and other sites. Provides central support to venue network & telecom teams in alignment with the Venue Technology Manager.

  • Cyber Security: responsible for the monitoring and management of all cyber security services including CSIRT, CSOC and interfacing with government cyber security agencies.  

  • Event Applications: responsible for the monitoring and management all event management and event specific applications, including coordination with remote application support groups.

  • Digital Services: responsible for the monitoring and management of web, mobile, ticketing and other digital services, including coordination with remote support groups.

  • Results, Timing & Scoring: responsible for supervising and coordinating the support at the venues and centrally for timing & scoring and results services. (e.g. on-venue results, digital graphics, etc)

  • Enterprise Services: responsible for supervising and coordinating the support for all organising committee applications and services. Sometimes these services are known as back-office IT or office & admin services. 

  • Infrastructure Services: responsible for the monitoring and management of all cloud services, on-site data centres / edge computing and related technical services such as active directory, NTP, Backup, OS, databases & middleware.

  • Technology Service Management: responsible for all service management activities including service desks, ITSM tools, operational change & release management and SLA monitoring and reporting. 

Prior to the event the TOC will be made operational on a smaller scale to support Test Events, Technology Rehearsals and bump-in activities. It will ramp up on a phased basis closer to the event until it is fully operational, and all positions are staffed. 

The number of seats and layout of the TOC is dependent on event needs and constraints. 

Prior to having a physical TOC facility, it is beneficial to establish an early-stage virtual TOC structure that uses on-call support for operational services. This enables the evolution and continuous improvement of the tools, processes and allows the Technology team to become familiar with these tools and processes well ahead of the event.

TOC shift patterns

Typically, the TOC is operational 24 x 7, including the service desk. For short events (such as multi-sport games) this may translate into full team coverage over 24 hours, while in other cases it may be more practical to have a skeleton crew working overnight to monitors services, perform maintenance activities and respond to any critical issues. 

The corresponding shift patterns depend on local working hour regulations, resource availability and needs. 

Some example shift patterns for 24 x 7 operations with approximately 40 working hours per week include:

  • 2 x 12 hour shift: (4 staff required if working 42 hours / week)

  • 3 x 8 hour shifts: (4 staff required if working 42 hours / week)

  • 2 x 8.5 hour primary shifts (giving coverage from 7:30am – 12:30am) with a 7 hour overnight skeleton shift

Including a skeleton shift, instead of full 24 x 7 shift coverage for all position can result in a significant reduction in resource needs. 

For example if 10% of TOC roles are required overnight, then having 2 x primary shifts and 1 x skeleton will result in a 20% reduction in quantity of resources needed. 

If 20% of roles are required overnight the it results in a 15% reduction in the quantity of resources

Need help with your resource plans? Let us know. 

TOC size and layout

The key question to be answered is:

“What portion of the technology operations / support team will physically be located in the same space?”

Excluding venue technology and service desk, there may still be thousands of people involved in the support model. Telecommunications monitoring, applications support, software and hardware vendors, etc. In an ideal world they should all be using the same processes, same tools and adhering to the same set of service levels. 

The minimum requirement for the TOC would be Duty Managers across each of the respective service groups. They would be in contact with their own support teams, but the specific technical resources would not be sitting in the TOC. This would serve as a management coordination and escalation function. So maybe 10 – 15 people in the TOC, each of whom is responsible for, and communication with a specific service team or teams. This could work however the gap would be a lack of cross-functional communications between technical experts in case of a critical incident. In many cases incidents require multiple technical teams (e.g. network, security, applications, infrastructure) to work together quickly to diagnose and resolve and incident. Having them all in different locations results in communications gaps and hence delays. 

The other extreme would be to have all technical teams in the same physical space. So, several hundred positions in the TOC. The issue here is available space and whether the benefits outweigh the costs. 

The explosion in the use of collaboration tools in 2020 due to Covid 19 has shown that critical operations can run effectively with people working remotely. So we think that the days of a large TOC with 100+ people is no longer necessary to ensure rapid team collaboration and issue resolution.

Alternate Technology Operations Centre

The alternate TOC is a business continuity solution that provides a backup location for TOC staff to work from in case the TOC is not usable. E.g. fire, earthquake, etc