Linux Systems Specialist - Senior | Tier 3

Senior Linux Systems Specialist - External Support (Level 3) Crafty Penguins | Full-time, permanent | Remote within Canada Compensation: $90,000–$120,000 CAD estimated annualized total, depending on experience, demonstrated capability, and breadth of skills. Linux is the common denominator, not the boundary of the job Crafty Penguins is a Linux, Open Source, DevOps, and infrastructure consulting company. We help organizations operate, troubleshoot, secure, improve, and scale the systems their businesses depend on. Our customers and their environments are extraordinarily diverse. We support SaaS companies, financial institutions, industrial clients, embedded and appliance vendors, internet service providers, and other organizations. Their systems include traditional servers running Nginx and PHP-FPM, cloud and Kubernetes platforms, virtualization and storage, and specialized Linux appliances and embedded devices. Some customers have modern, well-automated infrastructure; others have unusual, older, or heavily customized systems that still need to work reliably. We're looking for a senior, client-facing Linux specialist who can navigate that variety. Someone with deep Linux fundamentals, strong networking knowledge, enough software literacy to investigate production applications, and the judgment to solve unfamiliar problems independently. This isn't a role limited to following runbooks, patching servers, or escalating every issue that crosses the line between infrastructure and application code. What Level 3 means here Our levels describe capability and primary responsibilities, not rigid job boundaries. Everyone pitches in, whether that means routine maintenance or a difficult production incident. A Level 3 specialist is someone we trust to own and resolve a wide range of complex production problems with minimal supervision. You can enter an unfamiliar customer environment, establish how its components fit together, collect evidence, reason from first principles, and drive an issue to resolution. You recognize when specialized assistance is needed and involve colleagues with a clear account of what you've already established. Our Level 4 specialists are the people other specialists turn to for difficult technical guidance and mentoring. Our Level 5 specialists work at an exceptionally deep level, including designing or building new technologies and approaches. We generally grow Level 4 and Level 5 specialists internally, rather than hiring directly into those levels. We're not expecting you to arrive knowing everything. We are expecting the foundation, independence, and curiosity from which that deeper expertise can develop. Your technical foundation - required These are baseline competencies, not a list of keywords to put on a resume. We expect you to understand the underlying concepts and apply them in production, even when you need to look up a particular command option or configuration directive. Linux operating systems - Strong hands-on administration and troubleshooting across Debian/Ubuntu and RHEL/Rocky-style systems. - Understanding of processes, services, boot, permissions, authentication, filesystems, storage, memory, CPU, and I/O behaviour. - Ability to use logs, process inspection, system tracing, performance metrics, and other evidence to isolate failures. - Comfort investigating system behaviour beneath the level of a service restart or a copied troubleshooting procedure. IP networking and internet protocols - Solid, immediately accessible knowledge of IP addressing, subnetting, routing, switching fundamentals, DNS, NAT, firewalls, and common transport and application protocols. - Ability to investigate host and network connectivity with Linux networking tools and packet captures. - Understanding of how packets travel between applications, hosts, gateways, firewalls, and remote services, including how to locate failures along that path. - Experience diagnosing network behaviour rather than simply verifying whether a host answers ping. Web and application infrastructure - Working knowledge of HTTP/HTTPS, TLS, web servers, reverse proxies, load balancers, and application runtimes. - Experience across many web stacks, from older application servers and long-lived PHP, Java, and similar platforms to current reverse proxies, language runtimes, and frameworks. Nginx, Apache, and PHP-FPM are common examples, not the limit of what you may encounter. - Understanding of how applications depend on databases, caches, message queues, local storage, external APIs, and network services. - Ability to investigate configuration limits, timeouts, request handling, application errors, resource exhaustion, and performance problems across the full request path. Software literacy and automation - Confident shell scripting and useful experience with Python or another general-purpose programming language. - Ability to read unfamiliar application code, follow program flow, and interpret exceptions, stack traces, and logs. - Familiarity with APIs, common data formats, Git, and basic SQL. - Ability to build diagnostic tools or small automations, and make straightforward code or configuration fixes when appropriate. - Good judgment about testing, change safety, and avoiding unintended effects in production. You do not need to be a full-time software developer. You do need to understand enough about software to help determine why it is failing. The fact that a Linux host is running does not, by itself, mean your troubleshooting is finished. Production operations and reliability - Systematic troubleshooting: forming hypotheses, gathering evidence, isolating components, and verifying fixes. - Understanding of monitoring, alerting, logging, and interpreting reliability and performance metrics. - Familiarity with high availability, capacity, backups, recovery, and common production failure modes. - Ability to plan and execute safe changes, with appropriate testing, communication, and rollback procedures. - Interest in root causes and reducing repeat incidents through better monitoring, automation, documentation, and design. We value a Site Reliability Engineering mindset, even though this is not an engineering-titled position: restore service, understand the failure, and make the system more reliable afterward. Breadth of infrastructure experience You should have substantial practical experience in several of the areas below and enough general knowledge to become productive in others. We do not expect expert-level proficiency in every product we support. - Virtualization and storage: KVM, Proxmox, VMware, Xen/XCP-ng, LVM, RAID, NFS, iSCSI, ZFS, Ceph, or similar platforms. - Containers and orchestration: Docker, Podman, LXC/LXD, Kubernetes, or comparable tools; including workload, networking, and storage troubleshooting. - Cloud infrastructure: Practical production experience with at least one major provider, such as AWS, Azure, Google Cloud, DigitalOcean, or OVHcloud, and a working grasp of cloud networking, compute, and storage principles. - Databases and data services: MySQL/MariaDB, PostgreSQL, Redis, or comparable systems; including availability, connectivity, basic queries, and common performance issues. - Automation and delivery: SaltStack, Ansible, Terraform, CI/CD systems, or other tools used to build and maintain repeatable infrastructure. Specific technology experience matters, but strong transferable fundamentals and demonstrated problem-solving ability matter more. You're good at - Walking into an unfamiliar environment and learning how it works without needing a step-by-step guide. - Solving technical problems with the customer's business priorities, risk, and uptime in mind. - Taking responsibility for an investigation and communicating progress clearly. - Building trust with customers through sound judgment, clear explanations, and honest communication. - Collaborating with colleagues, asking for help appropriately, and sharing what you've learned. - Following established processes while noticing where those processes can be improved. - Managing your work, documentation, commitments, and time effectively in a remote consulting environment. - Staying curious and continually expanding both the breadth and depth of your technical skills. What you'll do - Investigate and resolve production incidents across Linux systems, networking, web applications, databases, containers, and infrastructure. - Respond to customer requests and system alerts, including incidents where the root cause is not immediately apparent. - Work with customer developers and technical teams to diagnose application and infrastructure issues. - Perform system maintenance, patching, health checks, deployments, migrations, and upgrades. - Onboard new customers, document their environments, and contribute to roadmaps for improving reliability and maintainability. - Develop scripts and automation that reduce recurring work, operational risk, or troubleshooting time. - Document technical findings, update customer assets and runbooks, and maintain accurate ticket and time records. - Participate in some scheduled after-hours customer maintenance when changes are best performed outside business hours. Participation in our after-hours on-call rotation and incident response is optional, by invitation, and separately compensated. This is distinct from occasional scheduled maintenance work. This role may not be the right fit if - Most of your Linux experience has been limited to following procedures, patching servers, and restarting services. - You are comfortable using tools but struggle to explain the computing or networking concepts behind them. - You routinely hand off production problems once they appear to involve application behaviour rather than the host operating system. - You need someone else to direct your investigation when you encounter an unfamiliar technology or failure. - You want to work exclusively in one narrow platform or product family. - You depend on documentation, search tools, or AI to supply fundamental concepts rather than using them to check details and accelerate work. There is nothing wrong with being at an earlier stage of your career or choosing a narrow specialty. This role simply calls for an established, broad technical foundation. Experience that would set you apart - Experience in Linux consulting, managed services, or other multi-customer environments. - Significant production SaaS experience, including observability, application performance, and high-availability operations. - Deep Kubernetes troubleshooting, complex virtualization, or distributed storage expertise. - Advanced routing, overlay networking, VPNs, or packet-level analysis. - Database replication, clustering, query optimization, or advanced performance tuning. - Software development experience beyond scripting, or contributions to open-source projects. - Low-level Linux experience involving kernel internals, eBPF, or specialized systems work. - Experience designing reliability improvements after difficult production incidents. These are differentiators, not requirements to already be operating at our Level 4 or Level 5. Experience and qualifications - Approximately five or more years of meaningful, hands-on Linux infrastructure experience, or equivalent demonstrated ability. - A record of independently investigating and resolving production problems. - Experience across multiple Linux-related infrastructure or application technologies. - Strong written and verbal communication skills for a customer-facing consulting role. Degrees and certifications are welcome, but demonstrated knowledge and practical capability carry more weight than credentials alone. Our interview process The first interview includes people from both HR and Engineering. We move back and forth between technical questions and questions about how you work with other people, so you are not in one kind of conversation for the whole time. We begin with a short technical screening to establish the foundational knowledge expected of a Level 3 specialist. We are not testing whether you have memorized every option in a manual. We are testing whether you already understand the essential concepts you would rely on when solving problems for our customers. In everyday work, using documentation, search, source code, colleagues, and AI tools is normal. In this screening, we want to establish the technical foundation you bring without those aids. After this first full interview, if we invite you back, the second interview is typically the final one. Why join Crafty Penguins? You'll work on an unusually broad mix of technologies and customer problems, alongside colleagues who enjoy difficult investigations and take pride in learning. You'll have autonomy, opportunities to grow into more specialized technical leadership, and a team willing to share knowledge. We are a remote-first organization with flexible scheduling, a culture of continuous improvement and infrastructure to support experimentation and learning. Our team is united by habits of building trust, looking at the bigger picture, leaning into challenges, and being better today than yesterday. Location, compensation, and application - Location: Fully remote within Canada. Crafty Penguins is based in Lethbridge, Alberta. - Employment: Full-time, permanent. - Compensation: Estimated annualized total compensation of $90,000–$120,000 CAD following successful completion of the three-month probationary period, based on a 40h work week. Placement in the range depends on experience, demonstrated capability, and breadth of skills. The stated range is an annualized post-probation package, not necessarily first-year earnings. - Eligibility: The successful applicant must be a Canadian citizen or permanent resident of Canada. Please submit your resume and cover letter, including the word Pepsi, and tell us briefly about a difficult production problem you personally investigated: what was happening, how you approached it, and what you discovered. Due to application volume, only shortlisted applicants will be contacted. Salary: $90K to $120K Originally posted on Himalayas