Number of valid parenthesis - algorithm

Question

I have to find out the number of valid parenthesis.Parenthesis are of two type [] ,(). How many ways are there to construct a valid sequence using X and Y number of [] ,(). For this problem we consider ([]) is invalid way i.e () can't hold [].
Is there any better solution than recursion.
For Example X=1 and Y=1
[]()
()[]
[()]

Answer 1

For any given combination of grouped, self-enclosed round parentheses and a given arrangement of square brackets, we can use the multiset combination to determine the number of arrangements:
n + k - 1 choose k, where k is the smaller of the number of self-enclosed parenthetical groups and the total number of square brackets, and n is the larger of the two + 1.
The number of arragements of square brackets is the nth Catalan number.
One way to generate the groups of parentheses is to assign an increasing number of groups, and calculate the number of distinct permutations for each partition of (X - number of assigned groups) multiplied by the sum of the parts-as-nth-Catalan. For example:
X = 4
Counts for each grouping:
1 group: Cat 3
2 groups: Cat 2 * 2 + 1 // partitions [2,0] * 2 and [1,1]
3 groups: 3 // partition [1,0,0] * 3
4 groups: 1 // partition [0,0,0,0]
I have yet to think of a way to avoid the partitions and would be interested to learn.

Answer 2

Because square brackets cannot be inside round brackets, every sequence of round brackets has to be complete (i.e. the count of the number of open minus close brackets is back down to zero) before each occurance of a square bracket.
1 012 1 0
()[(())()][[()()]((()))]()
10 121010 1010 123210 10
This effectively means that the square brackets split the round brackets into seperate independent sequences:
1 0 1 2 1 0
...[...]...[...[...]...]...
where complete sequences of round brackets can be placed anywhere between the square brackets.
As mentioned by Paul Hankin, the number of valid sequences of just one type of brackets is given by the Catalan Number. This gives you the number of valid sequences for the square brackets, and the number of valid sequences for any number of round brackets to be inserted somewhere inbetween the square brackets.
Unfortunately, finding out in what ways the round brackets can be distributed to the different locations inbetween the square brackets is a partitioning problem, and the logical way to solve this is by recursion. So this train of thought could lead to an algorithm, but not to one which avoids recursion.
However, the number of recursions needed to find partitions is much smaller than the Catalan number (e.g. 20 has 627 partitions, but the Catalan number for 20 is 6,564,120,420), so this method will be much faster than recursively enumerating every valid sequence.
Here's a simple example to demonstrate how this would work:
X=2, Y=3
The valid sequences of square brackets:
CatalanNumber(2) = 2
1. [ ] [ ]
2. [ [ ] ]
The positions for the round brackets:
2 × X + 1 = 5
1[2]3[4]5
1[2[3]4]5
The partitions of the round brackets:
p(Y) = 3
3
2,1
1,1,1
The permutations of the partitions:
partition: 3 (number of parts: 1)
→ 5 = 5
3,0,0,0,0 0,3,0,0,0 0,0,3,0,0 0,0,0,3,0 0,0,0,0,3
partition: 2,1 (number of parts: 2)
→ 5 × 4 = 20
2,1,0,0,0 2,0,1,0,0 2,0,0,1,0 2,0,0,0,1
1,2,0,0,0 0,2,1,0,0 0,2,0,1,0 0,2,0,0,1
1,0,2,0,0 0,1,2,0,0 0,0,2,1,0 0,0,2,0,1
1,0,0,2,0 0,1,0,2,0 0,0,1,2,0 0,0,0,2,1
1,0,0,0,2 0,1,0,0,2 0,0,1,0,2 0,0,0,1,2
partition: 1,1,1 (number of parts: 3, identical: 3)
→ (5 × 4 × 3) / (1 × 2 × 3) = 10
1,1,1,0,0 1,1,0,1,0 1,1,0,0,1
1,0,1,1,0 1,0,1,0,1 1,0,0,1,1
0,1,1,1,0 0,1,1,0,1 0,1,0,1,1 0,0,1,1,1
The valid sequences of round brackets:
CatalanNumber(3) = 5
1. ()()()
2. ()(())
3. (())()
4. (()())
5. ((()))
CatalanNumber(2) = 2
1. ()()
2. (())
CatalanNumber(1) = 1
1. ()
CatalanNumber(0) = 1
1.
The valid sequences per partition:
3,0,0,0,0 5 x 1 x 1 x 1 x 1 = 5
2,1,0,0,0 2 x 1 x 1 x 1 x 1 = 2
1,1,1,0,0 1 x 1 x 1 x 1 x 1 = 1
The total number of valid sequences:
Square brackets: 2
Round Brackets: 75
5 x 5 = 25
20 x 2 = 40
10 x 1 = 10
Total: 2 × 75 = 150