Skew Symplectic and Orthogonal Schur Functions

Using the vertex operator representations for symplectic and orthogonal Schur functions, we define two families of symmetric functions and show thatthey are the skew symplectic and skew orthogonal Schur polynomials defined implicitly by Koike and Terada and satisfy the general branching rules. Furthermore, we derive the Jacobi-Trudi identities and Gelfand-Tsetlin patterns for these symmetric functions. Additionally, the vertex operator method yields their Cauchy-type identities. This demonstrates that vertex operator representations serve not only as a tool for studying symmetric functions but also offers unified realizations for skew Schur functions of types A, C, and D.

With the advent of Schur symplectic and Schur orthogonal symmetric functions, arises the inquiry into skew versions for these symmetric functions.Since the family of symplectic Schur or orthogonal Schur functions indexed by partitions does not constitute a complete basis in the ring of symmetric functions (in fact they belong to the ring of Laurent polynomials), the traditional approach of employing the adjoint operation is not viable for defining skew orthogonal or symplectic Schur functions.Instead the skew symplectic and (even) skew orthogonal symmetric functions were implicitly defined by Koike and Terada [15] by restricting the irreducible characters of the classical groups into their subgroups and were expressed by tableaux representations.In [2], the tableaux representations were transformed into the Gelfand-Tsetlin pattern representations by Ayyer and Fischer.It is natural to ask whether the skew symplectic Schur and orthogonal Schur functions can be computed by Jacobi-Trudi type formulas, which would then lead to a parallel formulism as the Schur functions and skew Schur functions and also provide a lifting method to define the skew Schur symplectic and orthogonal functions in infinitely many variables.
The aim of this paper is to address these questions by the vertex operator realization of the classical symmetric functions [9] and the symplectic Schur and orthogonal Schur functions [3,12].The vertex algebraic method of studying Schur polynomials can be traced back to the Kyoto school's work on integrable systems [6] where the tau functions of the KP hierarchy are in terms of Schur polynomials.Skew Schur functions s λ/µ satisfy the celebrated decomposition formula, i.e., the general branching rule s λ (x 1 , . . ., x n ) = µ⊂λ s µ (x 1 , . . ., x n−k )s λ/µ (x n−k+1 , . . ., x n ), which can be interpreted as the restriction of the irreducible representation of GL(n, C) to GL(n − k, C) × GL(k, C).We will define skew (symplectic/orthogonal) Schur functions using vertex operators and derive their combinatorial properties such as the Jacobi-Trudi identities, Cauchy-type identities and verify they satisfy the Gelfand-Tsetlin pattern representations found by [2].We also show that our vertex operator representations for skew-type symmetric functions comply with the general branching rules in the agreement with Koike and Terada's approach.
The construction of skew Schur functions and their generalizations can be formulated in terms of the representation theory of infinite-dimensional Lie algebras.One starts with the infinitedimensional Heisenberg algebra [8] with the center 1 and defines the Fock space M * and its completion M * generated by the vacuum ⟨0| (which is the vacuum vector in the right module) and the Heisenberg generators with positive modes.There are three families of special elements (⟨λ|, ⟨λ sp | and ⟨λ o | in the right module) defined by partitions λ, corresponding respectively to the Schur, symplectic Schur, and orthogonal Schur cases.We will prove that they are orthonormal by computing the inner product with the elements (|λ⟩, |λ sp ⟩ and |λ o ⟩ in the left module) in the Fock space M respectively (see Theorem 2.3), which is key to construct (skew) symmetric functions and also savages the problem of adjoint operation with skew versions.For the half vertex operators Γ + ({x}) and Γ + x ± , we will show that where each summation is over (generalized) partitions λ = (λ 1 , . . ., λ N ) with zeros parts allowed in the end.The benefit of this approach is that we can easily obtain the well-known classical Cauchy identities for (symplectic/orthogonal) Schur functions [14,25].This approach is logically independent of the Jacobi-Trudi formula.
It is well known that ⟨µ|Γ + ({x})|λ⟩ = s λ/µ (x) [16,21] using the half vertex operator Γ + ({x}).We extend the vertex operator method to derive two families of symmetric functions which can be interpreted as the skew symplectic/orthogonal Schur functions from the general branching rule.We also show that each skew function can be written as a Jacobi-Trudi type determinant of complete homogeneous symmetric functions with variables x 1 , x −1 1 , . . ., x N , x −1 N .Exploiting the combinatorial properties of ⟨µ sp | and ⟨µ o |, we also obtain Gelfand-Tsetlin pattern representations (generating functions of some Young tableaux in an equivalent form [24]) for skew symplectic/orthogonal Schur functions, which provide alternative definitions of symplectic/orthogonal Schur functions (see [2]).The vertex operator method depending on the combinatorial properties of some orthogonal elements of the Fock space M * may shed light on specific descriptions of generalized shape π symmetric functions [7].
This paper is organized as follows.In Section 2, we recall the vertex operators related to fermionic Fock spaces M and M * and prove the orthonormality for three distinguished sets of M and the completion of M * .In Section 3, we define the skew symplectic/orthogonal Schur functions by the vertex algebraic method and derive their Jacobi-Trudi identities.We also show the vertex operator realizations for these skew symmetric functions are consistent with the general branching rule, therefore they agree with the skew symplectic and orthogonal Schur polynomials introduced by Koike and Terada.In Section 4, we give formulas of skew symplectic/orthogonal Schur functions in terms of Gelfand-Tsetlin pattern representations, which agree with Ayyer and Fischer's results.In Section 5, we provide a vertex operator approach to the Cauchy identities for the (symplectic/orthogonal) Schur functions, which is logically independent of the Jacobi-Trudi formulas [13].We also obtain Cauchy-type identities for skew (symplectic/orthogonal) Schur functions.

Preliminaries
Let H be the Heisenberg algebra generated by {a n | n ̸ = 0} with the central element c = 1 subject to the commutation relations [8] [a m , a n ] = mδ m,−n c, [a n , c] = 0. (2.1) The Fock space M (resp.M * ) is generated by the vacuum vector |0⟩ (resp.dual vacuum vector ⟨0|) and subject to In other words, M and M * are respectively left and right modules for H.It is easy to see that H acts on M (resp.M * ) irreducibly.Moreover, M (resp.
Note that M * is a graded space with the gradation induced from that of H. Let M * n be the filtration of the subspaces M * n spanned by homogeneous elements with degree ≥ n.Then we let M * be the associated completion of the Fock space M * .
Consider the following vertex operators given in [3,12] (see also [23]) 3) are obvious, and we only need to prove (2.4).Note the fact (2.5) Taking the coefficient of z n of (2.5), we have . Similarly, we can get the other commutation relations in (2.4).■ The following result can be found in [12, Theorem 3.4, Proposition 3.6].
Lemma 2.2.The components of the vertex operators (2.2) satisfy the following commutation relations ) is a weakly decreasing sequence of positive integers, and a generalized partition λ = (λ 1 , λ 2 , . . ., λ n ) is a weakly decreasing finite sequence of nonnegative integers.The λ i 's are called the parts of λ, and the number l(λ) of nonzero parts is the length of λ.If λ, µ are partitions, we will write µ ⊂ λ to mean that λ i ≥ µ i for all i ≥ 1.Therefore, a generalized partition is a partition appended with a string of finitely many zeros. 1 For a given (generalized) partition λ = (λ 1 , λ 2 , . . ., λ l ), let ) ) For any ⟨u| in M * and |v⟩ in M * , one define ⟨u|v⟩ by where it is assumed that ⟨0|1|0⟩ = 1.Now we have the following orthonormality result for three distinguished sets of M * .
Proof .Using the commutation relations (2.6), we have ), and thus it kills |0⟩ according to (2.3).If λ 1 > µ 1 , we move X −λ 1 −1 to the left to kill ⟨0|.Either way shows that the inner product is simplified to In the case of ⟨µ sp |λ sp ⟩, for µ 1 ≥ λ 1 we also have that (2.3).We therefore obtain (2.11) by combining the relations between µ i and λ i .The relation (2.12) can be proved similarly.■ ) is not bigger than l, then For example, it is easy to check that thus for µ = (4, 1, 0, 0) The same remark applies to the orthogonal case (2.12).

Skew symplectic/orthogonal Schur functions and Jacobi-Trudi identities
This section first recalls the vertex operator construction of skew Schur functions s λ/µ (x).Then we extend the method to realize skew symplectic Schur functions sp λ/µ x ± and skew orthogonal Schur functions o λ/µ x ± .Their Jacobi-Trudi formulas are also provided.The Jacobi-Trudi identities for the Schur function s λ (x), the symplectic Schur function sp λ x ± , and the orthogonal Schur function o λ x ± can be treated in the same manner by the vertex operator.Recall the classical Jacobi-Trudi identity for the Schur functions [19, formula (3.4)] where the complete homogeneous function h n (x) is defined by For the symplectic and the orthogonal cases, Schur function sp λ x ± and orthogonal Schur function o λ x ± admit the Jacobi-Trudi formulas [14, Theorems 1.3.2 and 1.3.3]and [12, formulas (4.1) and ( 4 where Introduce the (half) vertex operators

Skew symplectic Schur functions
It follows from (2.4) and (2.6) that In other words, if we replace the ith factor , it remains the same.We use the notation ( a b ) to mean either of a or b.Then for a generalized partition µ = (µ 1 , . . ., µ l ), Note that there is a similar action of S N on the vectors ).Therefore, we have that We now define a family of symmetric functions by vertex operators.
Proof .By definition of the vertex operator Y * (z), one has that where we have used the Vandermonde type identity [12, formula (4.6)] and [26, p. 229]: In view of (3.9) for partition ν = (ν 1 , . . ., ν N ), the coefficient of Then we have where the sum is over all generalized partitions ν = (ν 1 , . . ., ν N ) and we have used the bialternant formula of the symplectic Schur function sp ν x ± (see (1.2)) associated to the partition ν.Invoking (3.9) again and also by the Jacobi-Trudi formula for the symplectic Schur functions (3.2), we obtain another expression for ⟨0|Γ + x ± : By definition of vertex operator Y * (z), we have which implies It then follows from (2.8), (3.14) and (3.9) that where a ij are defined in (3.11)where the sum is over all generalized partitions η = (η 1 , . . ., η n ).
Proposition 3.9.For , the functions o λ/µ satisfy the general branching rule 4 Gelfand-Tsetlin pattern representations for three skew-type functions In this section, we first derive the formulas for skew Schur functions in Gelfand-Tsetlin patterns using vertex operators.We then do the same for skew symplectic/orthogonal Schur functions.We start by recalling a special case of skew Schur functions in terms of vertex operators [11,Proposition 2.4].We say a (generalized) partition λ interlaces another (generalized) partition ν, denoted as ν ≺ λ, if λ i ≥ ν i ≥ λ i+1 for all i.

Cauchy-type identities and general branching rules
In this section, we give a new proof of Cauchy identities for (symplectic/orthogonal) Schur functions by the method of vertex operators.This approach is logically independent of the Jacobi-Trudi formula.The method can be further extended to skew Schur functions as well as skew symplectic/orthogonal Schur functions.
The following is a generalization of Lemma 5.1.
We now offer a vertex algebraic proof of the Cauchy-type identity for skew Schur functions [19, p. 93], which can be seen as a generalized Cauchy identity for Schur functions.
Theorem 5.7.For two partitions λ and µ, where the sum is over all partitions ρ and τ such that the skew Schur functions are defined.
Proof .From (2.10), we can easily deduce that where the sum is over all partitions ρ.Thus where we have used (3.6).We also have the following relation: We also extend the vertex algebraic approach to derive the Cauchy-type identities for skew symplectic/orthogonal Schur functions.sp λ/τ x ± S * µ/τ (y). (5.12) Comparing (5.11) with (5.12), we can get (5.9).The relation (5.10) can be proved similarly.■ Remark 5.9.The generalized partition τ in the right-hand side of (5.12) is contained in the generalized partition λ.For µ = 0 N , λ = 0 N , relation (5.12) recovers the classical Cauchy identity for Schur functions Remark 5.10.In [16], Lam studied some general symmetric functions via a boson-fermion correspondence.It will be interesting to study and realize the skew symmetric functions using vertex operators.
Added in proof.In the current paper posted on arXiv in 2022, among other things we have derived the Jacobi-Trudi formulas for the skew Schur symplectic functions and the skew Schur orthogonal functions in terms of the (generalized) homogeneous symmetric functions.We recently noticed that [1] has found a combinatorial proof of the alternative Jacobi-Trudi formulas in terms of elementary symmetric functions in 2023.We will also give a vertex algebraic proof of the latter in a forthcoming paper.

A Another proof of Proposition 3.1
In this appendix, we give a simple vertex algebraic proof of Proposition 3.1 and (5.5).