Expression.h

./expression.h consists of the definitions of abstract expressions and their related expression templates.

• Type
• Exp
• ScalarExp: public Exp<ScalarExp<DType>, DType, type::kMapper>
• TypecastExp: public Exp<TypecastExp<DstDType, SrcDType, EType, etype>, DstDType, etype>
• TransposeExp: public Exp<TransposeExp<EType, DType>, DType, type::kChainer>
• ExpEngine
• RValueExp: public Exp<Container, DType, type::kRValue>
• DotExp: public Exp<DotExp<TA, TB, ltrans, rtrans, DType>, DType, type::kComplex>
• TernaryMapExp: public Exp<TernaryMapExp<OP, TA, TB, TC, DType, etype>, DType, etype>
• UnaryMapExp: public Exp<UnaryMapExp<OP, TA, DType, etype>, DType, etype>
• BinaryMapExp: public Exp<BinaryMapExp<OP, TA, TB, DType, etype>, DType, etype>

Type

Every expression in MShadow has to own its own type.

• kRValue: this expression directly correspnds to a data class, can be used to assign data.
• kMapper: this expression contains element-wise tensor operations, can map a expression to same shape.
• kChainer: this expression can be chained with other expressions. Most of functions in file ./extension/ are of this type.
• kComplex: othercase: e.g dot product.

namespace type {
// type expression type are defined as bitmask
// subtype relationshop kRValue < kMapper < kPull < kComplex

const int kRValue = 0;
const int kMapper = 1;
const int kChainer = 3;
const int kComplex = 7;
} 

Exp

The struct Exp is the base class for every expression.

Each class inheritated from Exp has to put their type into SubType, while DType is the type of data being manipulated and exp_type is to indicate the expression type inheritated from it.

template<typename SubType, typename DType, int exp_type>
struct Exp {
 public:
  // return subtype instance of current class
  inline const SubType& self(void) const {
    return *static_cast<const SubType*>(this);
  }
  // return reference of subtype instance of current class
  inline SubType* ptrself(void) {
    return static_cast<SubType*>(this);
  }
};

ScalarExp

: public Exp<ScalarExp<DType>, DType, type::kMapper>

The ScalarExp is to map a scalar type to a sub-class ScalarExp of expression type Exp.

For now, we can just think explicit constructor is majorly used for avoiding the implicit consturction of class in declaration. the reason we cannot make the constructor explicit here is that, if the constructor is explicit, the occurence of scalar in either its lhs or rhs will not call this constructor.

template<typename DType>
struct ScalarExp: public Exp<ScalarExp<DType>, DType, type::kMapper> {
  // scalar value
  DType scalar_;
  // implicit constructor, MUST NOT BE explicit 
  ScalarExp(DType scalar) : scalar_(scalar) {}  
};

// this is a type cast function that turns the scalar s with type Dtype into type ScalarExp
template<typename DType>
inline ScalarExp<DType> scalar(DType s) {
  return ScalarExp<DType>(s); // turn the scalar s with type Dtype into type ScalarExp
}

// usage
scalar<float>(10.0f)
// it is worthy noting that this process is generally implicitly performed 
// (by overloaded operator in struct RValueExp)

TypecastExp

: public Exp<TypecastExp<DstDType, SrcDType, EType, etype>, DstDType, etype>

The TypecastExp is to explicitly change the DType of a expression type Exp.

template<typename DstDType, typename SrcDType, typename EType, int etype>
struct TypecastExp:
      public Exp<TypecastExp<DstDType, SrcDType, EType, etype>, DstDType, etype> {
  // expression to be typecasted
  const EType &exp;
  // constructor
  explicit TypecastExp(const EType &e) : exp(e) {}
};
// create an Typecast expression
template<typename DstDType, typename SrcDType, typename EType, int etype>
inline TypecastExp<DstDType, SrcDType, EType, (etype|type::kMapper)>
tcast(const Exp<EType, SrcDType, etype> &exp) {
  return TypecastExp<DstDType, SrcDType, EType, (etype|type::kMapper)>(exp.self());
}

// usage
float data[10];
int data1[10];

Tensor<cpu,2> mat(data,Shape2(5,2));
Tensor<cpu,2,int> mat1(data1,Shape2(5,2));

mat = 3.2f;
mat1 = tcast<int>(mat);

for (index_t i = 0; i < mat.size(0); ++i) {
  for (index_t j = 0; j < mat.size(1); ++j) {
      printf("%.2f ", mat[i][j]);
  }
  printf("\n");
}
for (index_t i = 0; i < mat1.size(0); ++i) {
  for (index_t j = 0; j < mat1.size(1); ++j) {
      printf("%d ", mat1[i][j]);
  }
  printf("\n");
}
// output:
// 3.20 3.20
// 3.20 3.20
// 3.20 3.20
// 3.20 3.20
// 3.20 3.20
// 3 3
// 3 3
// 3 3
// 3 3
// 3 3

TransposeExp

: public Exp<TransposeExp<EType, DType>, DType, type::kChainer>

The TransposeExp is generated by .T() function in class RValueExp.

Through the constructor function, a same type Tensor is generated.

The actual transpose actually happens in the `Eval()` function.
Whether its member function `T()` is useful or not remains to be checking.

template<typename EType, typename DType>
struct TransposeExp: public Exp<TransposeExp<EType, DType>,
                                DType, type::kChainer> {
  // expression to be transposed, the generated one is same
  const EType &exp;
  // constructor
  explicit TransposeExp(const EType &e) : exp(e) {}
  // transpose expression
  inline const EType &T(void) const {
    return exp;
  }
};

ExpEngine

ExpEngine is the struct that starts to interpret all of the expressions.

It is generated by member functions in RValueExp, while its implementation is in ./expr_engine-inl.h , which call the MapExp in ./tensor_cpu-inl.h or ./tensor_gpu-inl.h to begin the interpretion of expressions.

The details of its discussion is left to its implementation codes.

template<typename Saver, typename RValue, typename DType>
struct ExpEngine;

RValueExp

: public Exp<Container, DType, type::kRValue>

RValueExp is a special class, that is inheritated by TRValue (Tensor Rvalue), which is the super class of all kinds of tensors.

Since it explicitly set the exp_type (expression type) in Exp to kRValue during inheritance, it correspnds to every situation with a form dst () src, where dst is the destination to receive the result of computation, () is += / -= / *= / /= / =, and src can be a value or expression.

As a result, in this class, we have eight reloaded operator and two assign functions, with a extra transpose function.

template<typename Container, typename DType>
class RValueExp: public Exp<Container, DType, type::kRValue> {
 public:
  // transpose of a matrix, return transpose of current expression
  // usage:  *.T()
  inline const TransposeExp<Container, DType> T(void) const {
    return TransposeExp<Container, DType>(this->self());
  }
  // operator overload
  // += for scalar
  inline Container &operator+=(DType s) {
    ExpEngine<sv::plusto, Container, DType>::Eval(this->ptrself(), scalar<DType>(s));
    return *(this->ptrself());
  }
  // -= for scalar 
  inline Container &operator-=(DType s) {
    ExpEngine<sv::minusto, Container, DType>::Eval(this->ptrself(), scalar<DType>(s));
    return *(this->ptrself());
  }
  // *= for scalar
  inline Container &operator*=(DType s) {
    ExpEngine<sv::multo, Container, DType>::Eval(this->ptrself(), scalar<DType>(s));
    return *(this->ptrself());
  }
  // /= for scalar
  inline Container &operator/=(DType s) {
    ExpEngine<sv::divto, Container, DType>::Eval(this->ptrself(), scalar<DType>(s));
    return *(this->ptrself());
  }
  // assign for scalar
  inline Container &__assign(DType s) {
    ExpEngine<sv::saveto, Container, DType>::Eval(this->ptrself(), scalar<DType>(s));
    return *(this->ptrself());
  }
  // assign for expression
  template<typename E, int etype>
  inline Container &__assign(const Exp<E, DType, etype> &exp) {
    ExpEngine<sv::saveto, Container, DType>::Eval(this->ptrself(), exp.self());
    return *(this->ptrself());
  }
  // not sure the difference to above function
  inline Container &__assign(const Exp<Container, DType, type::kRValue> &exp);
  // += for expression
  template<typename E, int etype>
  inline Container &operator+=(const Exp<E, DType, etype> &exp) {
    ExpEngine<sv::plusto, Container, DType>::Eval(this->ptrself(), exp.self());
    return *(this->ptrself());
  }
  // -= for expression
  template<typename E, int etype>
  inline Container &operator-=(const Exp<E, DType, etype> &exp) {
    ExpEngine<sv::minusto, Container, DType>::Eval(this->ptrself(), exp.self());
    return *(this->ptrself());
  }
  // *= for expression
  template<typename E, int etype>
  inline Container &operator*=(const Exp<E, DType, etype> &exp) {
    ExpEngine<sv::multo, Container, DType>::Eval(this->ptrself(), exp.self());
    return *(this->ptrself());
  }
  // /= for expression
  template<typename E, int etype>
  inline Container &operator/=(const Exp<E, DType, etype> &exp) {
    ExpEngine<sv::divto, Container, DType>::Eval(this->ptrself(), exp.self());
    return *(this->ptrself());
  }
};

In every reloaded function, ExpEngine is used to trigger the Eval function, which use template MapExp to MakePlan as described above.

At last, the namespace sv is defined in the ./base.h to perform the final assignment.

DotExp

: public Exp<DotExp<TA, TB, ltrans, rtrans, DType>, DType, type::kComplex>

The DotExp is an expression to do matrix computation between two Tensors.

template<typename TA, typename TB, bool ltrans, bool rtrans, typename DType>
struct DotExp: public Exp<DotExp<TA, TB, ltrans, rtrans, DType>, DType, type::kComplex> {
  const TA &lhs_;
  const TB &rhs_;
  DType scale_; // scale over result

  explicit DotExp(const TA &lhs, const TB &rhs, DType scale)
      : lhs_(lhs), rhs_(rhs), scale_(scale) {}
};

The generation of struct DotExp is triggered by the function dot() with following four different styles by consideration of transpose.

• both lhs and rhs are without transpose

template<typename TA, typename TB, typename DType>
inline DotExp<TA, TB, false, false, DType>
dot(const RValueExp<TA, DType> &lhs, const RValueExp<TB, DType> &rhs) {
  return DotExp<TA, TB, false, false, DType>(lhs.self(), rhs.self(), DType(1.0f));
}

• lhs is with transpose, while rhs is not

template<typename TA, typename TB, typename DType>
inline DotExp<TA, TB, true, false, DType>
dot(const TransposeExp<TA, DType> &lhs, const RValueExp<TB, DType> &rhs) {
  return DotExp<TA, TB, true, false, DType>(lhs.exp, rhs.self(), DType(1.0f));
}

• rhs is with transpose, while lhs is not

template<typename TA, typename TB, typename DType>
inline DotExp<TA, TB, false, true, DType>
dot(const RValueExp<TA, DType> &lhs, const TransposeExp<TB, DType> &rhs) {
  return DotExp<TA, TB, false, true, DType>(lhs.self(), rhs.exp, DType(1.0f));
}

• both lhs and rhs are with transpose

template<typename TA, typename TB, typename DType>
inline DotExp<TA, TB, true, true, DType>
dot(const TransposeExp<TA, DType> &lhs, const TransposeExp<TB, DType> &rhs) {
  return DotExp<TA, TB, true, true, DType>(lhs.exp, rhs.exp, DType(1.0f));
}

the usage of batch_dot is unclear yet!

template<bool transpose_left, bool transpose_right, typename TA, typename TB, typename DType>
inline DotExp<TA, TB, transpose_left, transpose_right, DType>
batch_dot(const RValueExp<TA, DType> &lhs, const RValueExp<TB, DType> &rhs) {
  return DotExp<TA, TB, transpose_left, transpose_right, DType>(
    lhs.self(), rhs.self(), DType(1.0f));
}

TernaryMapExp

public Exp<TernaryMapExp<OP, TA, TB, TC, DType, etype>, DType, etype>

TernaryMapExp is designed to handle the ternary operation expression. Its member variables contain three different expressions as their original types.

template<typename OP, typename TA, typename TB, typename TC, typename DType, int etype>
struct TernaryMapExp: public Exp<TernaryMapExp<OP, TA, TB, TC, DType, etype>, DType, etype> {
  const TA &item1_;
  const TB &item2_;
  const TC &item3_;

  explicit TernaryMapExp(const TA &item1, const TB &item2, const TC &item3)
      :item1_(item1), item2_(item2), item3_(item3) {}
};

The struct TernaryMapExp is generated by the function MakeExp.

template<typename OP, typename TA, typename TB, typename TC, typename DType, int ta, int tb, int tc>
inline TernaryMapExp<OP, TA, TB, TC, DType, (ta|tb|tc|type::kMapper)>
MakeExp(const Exp<TA, DType, ta> &item1, const Exp<TB, DType, tb> &item2,
  const Exp<TC, DType, tc> &item3) {
  return TernaryMapExp<OP, TA, TB, TC, DType,
                      (ta|tb|tc|type::kMapper)>(item1.self(), item2.self(), item3.self());
}

The function F<op>() provides a short hand for MakeExp(), with a operator defined in namespace op, which provides Map() function to guide the way of evaluation.

template<typename OP, typename TA, typename TB, typename TC, typename DType, int ta, int tb, int tc>
inline TernaryMapExp<OP, TA, TB, TC, DType, (ta|tb|tc|type::kMapper)>
F(const Exp<TA, DType, ta> &item1, const Exp<TB, DType, tb> &item2,
 const Exp<TC, DType, tc> &item3) {
  return MakeExp<OP>(item1, item2, item3);
}

UnaryMapExp

public public Exp<UnaryMapExp<OP, TA, DType, etype>, DType, etype>

UnaryMapExp is designed to handle the unary operation expression. Its member variables contain only one expression as its original type.

template<typename OP, typename TA, typename DType, int etype>
struct UnaryMapExp: public Exp<UnaryMapExp<OP, TA, DType, etype>, DType, etype> {
  const TA &src_;
  explicit UnaryMapExp(const TA &src) : src_(src) {}
};

The struct UnaryMapExp is also generated by the function MakeExp.

template<typename OP, typename TA, typename DType, int ta>
inline UnaryMapExp<OP, TA, DType, (ta|type::kMapper)>
MakeExp(const Exp<TA, DType, ta> &src) {
  return UnaryMapExp<OP, TA, DType, (ta|type::kMapper)>(src.self());
}

Similarly, UnaryMapExp also has its own function F<op>() providing a short hand for MakeExp(), with a operator defined in namespace op, which provides Map() function to guide the way of evaluation.

template<typename OP, typename TA, typename DType, int ta>
inline UnaryMapExp<OP, TA, DType, (ta|type::kMapper)>
F(const Exp<TA, DType, ta> &src) {
  return MakeExp<OP>(src);
}

BinaryMapExp

: public Exp<BinaryMapExp<OP, TA, TB, DType, etype>, DType, etype>

BinaryMapExp is designed to handle the binary operation expression. Its member variables contain two different expressions as their original type.

template<typename OP, typename TA, typename TB, typename DType, int etype>
struct BinaryMapExp: public Exp<BinaryMapExp<OP, TA, TB, DType, etype>,
                                DType, etype> {
  const TA &lhs_;
  const TB &rhs_;
  explicit BinaryMapExp(const TA &lhs, const TB &rhs): lhs_(lhs), rhs_(rhs) {}
};

The struct BinaryMapExp is also generated by the function MakeExp.

template<typename OP, typename TA, typename TB, typename DType, int ta, int tb>
inline BinaryMapExp<OP, TA, TB, DType, (ta|tb|type::kMapper)>
MakeExp(const Exp<TA, DType, ta> &lhs, const Exp<TB, DType, tb> &rhs) {
  return BinaryMapExp<OP, TA, TB, DType, (ta|tb|type::kMapper)>(lhs.self(), rhs.self());
}

Similarly, BinaryMapExp also has its own function F<op>() providing a short hand for MakeExp(), with a operator defined in namespace op, which provides Map() function to guide the way of evaluation.

template<typename OP, typename TA, typename TB, typename DType, int ta, int tb>
inline BinaryMapExp<OP, TA, TB, DType, (ta|tb|type::kMapper)>
F(const Exp<TA, DType, ta> &lhs, const Exp<TB, DType, tb> &rhs) {
  return MakeExp<OP>(lhs, rhs);
}

Moreover, the four basic operators + / - / * / / are also overloaded to make sure we can add an Exp type to built-in scalar type, e.g. a+3.0f where a is a Tensor. Since the constructor of ScalarExp is not explicit, the 3.0f will be implicitly converted to type ScalarExp.

template<typename TA, typename TB, typename DType, int ta, int tb>
inline BinaryMapExp<op::plus, TA, TB, DType, (ta|tb|type::kMapper)>
operator+(const Exp<TA, DType, ta> &lhs, const Exp<TB, DType, tb> &rhs) {
  return MakeExp<op::plus>(lhs, rhs);
}

template<typename TA, typename TB, typename DType, int ta, int tb>
inline BinaryMapExp<op::minus, TA, TB, DType, (ta|tb|type::kMapper)>
operator-(const Exp<TA, DType, ta> &lhs, const Exp<TB, DType, tb> &rhs) {
  return MakeExp<op::minus>(lhs, rhs);
}

template<typename TA, typename TB, typename DType, int ta, int tb>
inline BinaryMapExp<op::mul, TA, TB, DType, (ta|tb|type::kMapper)>
operator*(const Exp<TA, DType, ta> &lhs, const Exp<TB, DType, tb> &rhs) {
  return MakeExp<op::mul>(lhs, rhs);
}

template<typename TA, typename TB, typename DType, int ta, int tb>
inline BinaryMapExp<op::div, TA, TB, DType, (ta|tb|type::kMapper)>
operator/(const Exp<TA, DType, ta> &lhs, const Exp<TB, DType, tb> &rhs) {
  return MakeExp<op::div>(lhs, rhs);
}